non response bias in medical research

An official website of the United States government

The .gov means it's official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you're on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Publications
Account settings
Browse Titles

NCBI Bookshelf. A service of the National Library of Medicine, National Institutes of Health.

StatPearls [Internet]. Treasure Island (FL): StatPearls Publishing; 2024 Jan-.

StatPearls [Internet].

Aleksandar Popovic ; Martin R. Huecker .

Affiliations

Last Update: June 20, 2023 .

Definition/Introduction

Bias is colloquially defined as any tendency that limits impartial consideration of a question or issue. In academic research, bias refers to a type of systematic error that can distort measurements and/or affect investigations and their results. [1] It is important to distinguish a systematic error, such as bias, from that of random error. Random error occurs due to the natural fluctuation in the accuracy of any measurement device, the innate differences between humans (both investigators and subjects), and by pure chance. Random errors can occur at any point and are more difficult to control. [2] Systematic errors, referred to as bias from here on, occur at one or multiple points during the research process, including the study design, data collection, statistical analysis, interpretation of results, and publication process. [3]

However, interpreting the presence of bias involves understanding that it is not a dichotomous variable, where the results can either be “present” or “not present.” Rather, it must be understood that bias is always present to some degree due to inherent limitations in research, its design, implementation, and ethical considerations. [4] Therefore, it is instead crucial to evaluate how much bias is present in a study and how the researchers attempted to minimize any sources of bias. [5] When evaluating for bias, it is important to note there are many types with several proposed classification schemes. However, it is easiest to view bias based on the various stages of research studies; the planning and design stage (before), data collection and analysis (during), and interpretation of results and journal submission (after).

Issues of Concern

The planning stage of any study can have bias present in both study design and recruitment of subjects. Ideally, the design of a study should include a well-defined outcome, population of interest, and collection methods before implementation and data collection. The outcome, for example, response rates to a new medication, should be precisely agreed upon. Investigators may focus on changes in laboratory parameters (such as a new statin reducing LDL and total cholesterol levels) or focus on long-term morbidity and mortality (does the new statin cause reduction in cardiovascular-related deaths?) Similarly, the investigator’s own pre-existing notion or personal beliefs can influence the question being asked and the study's methodology. [6]

For example, an investigator who works for a pharmaceutical company may address a question or collect data most likely to produce a significant finding supporting the use of the investigational medication. Thus, if possible, the question(s) being asked and the collection methods employed should be agreed upon by multiple team members in an interprofessional setting to reduce potential bias. Ethics committees also play a valuable role here.

Relatedly, the team members designing a study must define their population of interest, also referred to as the study population. Bias occurs if the study population does not closely represent a target population due to errors in study design or implementation, termed selection bias. Sampling bias is one form of selection bias and typically occurs if subjects were selected in a non-random way. It can also occur if the study requires subjects to be placed into cohorts and if those cohorts are significantly different in some way. This can lead to erroneous conclusions and significant findings. Randomization of subject selection and cohort assignment is a technique used in study design intended to reduce sampling bias. [7] [8]

However, bias can occur if subject selection occurred through limited means, such as recruiting subjects through phone landlines, thereby excluding anyone who does not own a landline. Similarly, this can occur if subjects are recruited only through email or a website. This can result in confounding or the introduction of 3 variable that influences both the independent and dependent variables. [9]

For example, if a study recruited subjects from two primary care clinics to compare diabetes screening and treatment rates but did not account for potentially different socioeconomic characteristics of the two clinics, there may be significant differences between groups not due to clinical practice but rather cohort composition.

A subtype of selection bias, admission bias (also referred to as Berkson bias), occurs when the selected study population is derived from patients within hospitals or certain specialty clinics. This group is then compared to a non-hospitalized group. This predisposes to bias as hospitalized patient populations are more likely to be ill and not represent the general population. Furthermore, there are typically other confounding variables or covariates that may skew relationships between the intended dependent and independent variables. [10]

For example, in one study that evaluated the effect of cigarette smoking and its association with bladder cancer, researchers decided to use a hospital-based case-control study design. Normally, there is a strong and well-established relationship between years of cigarette use and the likelihood of developing bladder cancer. In fact, part of screening guidelines for bladder cancer considers the total years that an individual has smoked during patient risk stratification and subsequent evaluation and follow-up. However, in one study, researchers noted no significant relationship between smoking and bladder cancer. Upon re-evaluating, they noted their cases and controls both had significant smoking histories, thereby blurring any relationships. [11]

Admission bias can be reduced by selecting appropriate controls and being cognizant of the potential introduction of this bias in any hospital-based study. If this is not possible to do, researchers must be transparent about this in their work and may try to use different methods of statistical analysis to account for any confounding variables. In an almost opposite fashion, another source of potential error is a phenomenon termed the healthy worker effect. The healthy worker effect refers to the overall improved health and decreased mortality and morbidity rates of those employed relative to the unemployed. This occurs for various reasons, including access to better health care, improved socioeconomic status, the beneficial effects of work itself, and those who are critically ill or disabled are less likely to find employment. [12] [13]

Two other important forms of selection bias are lead-time bias and length time bias. Lead-time bias occurs in the context of disease diagnosis. In general, it occurs when new diagnostic testing allows detection of a disease in an early stage, causing a false appearance of longer lifespan or improved outcomes. [14] An example of this is noted in individuals with schizophrenia with varying durations of untreated psychosis. Those with shorter durations of psychosis relative to longer durations typically had better psychosocial functioning after admission to and treatment within a hospital. However, upon further analysis, it was found that it was not the duration of psychosis that affected psychosocial functioning. Rather, the duration of psychosis was indicative of the stage of the person’s disease, and those individuals with shorter durations of psychosis were in an earlier stage of their disease. [15]

Length time bias is similar to lead-time bias; however, it refers to the overestimation of an individual’s survival time due to a large number of cases that are asymptomatic and slowly progressing with a smaller number of cases that are rapidly progressive and symptomatic. An example can be noted in patients with hepatocellular carcinoma (HCC). Those who have HCC found via asymptomatic screening typically had a tumor doubling time of 100 days. In contrast, those individuals who had HCC uncovered due to symptomatic presentation had a tumor doubling time of 42 days on average. However, overall outcomes were the same amongst these two groups. [16]

The effect of both lead time and length time bias must be taken into effect by investigators. For lead-time bias, investigators can instead look at changes in the overall mortality rate due to disease. One method involves creating a modified survival curve that considers possible lead-time bias with the new diagnostic or screening protocols. [17] This involves an estimate of the lead time bias and subsequently subtracting this from the observed survival time. Unfortunately, the consequences of length time bias are difficult to mitigate, but investigators can minimize their effects by keeping individuals in their original groups based on screening protocols (intention-to-screen) regardless of the individual required earlier diagnostic workup due to symptoms.

Channeling and procedure bias are other forms of selection bias that can be encountered and addressed during the planning stage of a study. Channeling bias is a type of selection bias noted in observational studies. It occurs most frequently when patient characteristics, such as age or severity of illness, affect cohort assignment. This can occur, for example, in surgical studies where different interventions carry different levels of risk. Surgical procedures may be more likely to be carried out on patients with lower levels of periprocedural risk who would likely tolerate the event, whereas non-surgical interventions may be reserved for patients with higher levels of risk who would not be suitable for a lengthy procedure under general anesthesia. [18] As a result, channeling bias results in an imbalance of covariates between cohorts. This is particularly important when the surgical and non-surgical interventions have significant differences in outcome, making it difficult to ascertain if the difference is due to different interventions or covariate imbalance. Channeling bias can be accounted for through the use of propensity score analysis. [19]

Propensity scores are the probability of receiving one intervention over another based on an individual's observed covariates. These scores are obtained through a variety of different methods and then accounted for in the analysis stage via statistical methods, such as logistic regression. In addition to channeling bias, procedure bias (administration bias) is a similar form of selection bias, where two cohorts receive different levels of treatment or are administered similar treatments or interviews in different formats. An example of the former would be two cohorts of patients with ACL injuries. One cohort received strictly supervised physical therapy 3 times per week, and the other cohort was taught the exercises but instructed to do them at home on their own. An example of the latter would be administering a questionnaire regarding eating disorder symptoms. One group was asked in-person in an interview format, and the other group was allowed to take the questionnaire at home in an anonymous format. [20]

Either form of procedure bias can lead to significant differences observed between groups that might not exist where they are treated the same. Therefore, both procedure and channeling bias must be considered before data collection, particularly in observational or retrospective studies, to reduce or eliminate erroneous conclusions that are derived from the study design itself and not from treatment protocols.

Bias in Data Collection & Analysis

There are also a variety of forms of bias present during data collection and analysis. One type is observer bias, which refers to any systematic difference between true and recorded values due to variation in the individual observer. This form of bias is particularly notable in studies that require investigators to record measurements or exposures, particularly if there is an element of subjectiveness present, such as evaluating the extent or color of a rash. [21] However, this has even been noted in the measurement of subjects’ blood pressures when using sphygmomanometers, where investigators may round up or down depending on their preconceived notions about the subject. Observer bias is more likely when the observer is aware of the subject’s treatment status or assignment cohort. This is related to confirmation bias, which refers to a tendency to search for or interpret information to support a pre-existing belief. [22]

In one prominent example, physicians were asked to estimate blood loss and amniotic fluid volume in pregnant patients currently in labor. By providing additional information in the form of blood pressures (hypotensive or normotensive) to the physicians, they were more likely to overestimate blood loss and underestimate amniotic fluid volume when told the patient was hypotensive. [23] Similar findings are noted in fields such as medicine, health sciences, and social sciences, illustrating the strong and misdirecting influence of confirmation bias on the results found in certain studies. [22] [24]

Investigators and data collectors need to be trained to collect data in a uniform, empirical fashion and be conscious of their own beliefs to minimize measurement variability. There should be standardization of data collection to reduce inter-observer variance. This may include training all investigators or analysts to follow a standardized protocol, use standardized devices or measurement tools, or use validated questionnaires. [21] [25]

Furthermore, the decision of whether to blind the investigators and analysts should also be made. If implemented, blinding of the investigators can reduce observer bias, which refers to the differential assessment of an outcome when subjective criteria are being assessed. Confirmation bias within investigators and data collectors can be minimized if they are informed of its potential interfering role. Furthermore, overconfidence in either the overall study’s results or the collection of accurate data from subjects can be a strong source of confirmation bias. Challenging overconfidence and encouraging multiple viewpoints is another mechanism by which to challenge this within investigators. Lastly, potential funding sources or other conflicts of interest can influence confirmation and observer bias and must be considered when evaluating for these potential sources of systematic error. [26] [27] However, subjects themselves may change their behavior, consciously or unconsciously, in response to their awareness of being observed or being assigned to a treatment group termed the Hawthorne effect. [28] The Hawthorne effect can be minimized, although not eliminated, by reducing or hiding the observation of the subject if possible. A similar phenomenon is noted with self-selection bias, which occurs when individuals sort themselves into groups or choose to enroll in studies based on pre-existing factors. For example, a study evaluating the effectiveness of a popular weight loss program that allows participants to self-enroll may have significant differences between groups. In circumstances such as this, it is more probable that individuals who experienced greater success (measured in terms of weight lost) are likely to enroll. Meanwhile, those who did not lose weight and/or gained weight would likely not enroll. Similar issues plague other studies that rely on subject self-enrollment. [20] [29]

Self-selection bias is often found in tandem with response bias, which refers to subjects inaccurately answering questions due to various influences. [30] This can be due to question-wording, the social desirability of a certain answer, the sensitiveness of a question, the order of questions, and even the survey format, such as in-person, via telephone, or online. [22] [31] [32] [33] [34] There are methods of reducing the impact of all these factors, such as the use of anonymity in surveys, the use of specialized questioning techniques to reduce the impact of wording, and even the use of nominative techniques where individuals are asked about the behavior of close friends for certain types of questions. [35] Non-response bias refers to significant differences between individuals who respond and those who do not respond to a survey or questionnaire. It is not to be confused as being the opposite of response bias. It is particularly problematic as errors can result in estimating population characteristics due to a lack of response from the non-responders. It is often noted in health surveys regarding alcohol, tobacco, or drug use, though it has been seen in many other topics targeted by surveys. [36] [37] [36] Furthermore, particularly in surveys designed to evaluate satisfaction after an intervention or treatment, individuals are much more likely to respond if they felt highly satisfied relative to the average individual. While highly dissatisfied individuals were also more likely to respond relative to average, they were less likely to respond relative to highly satisfied individuals, thus potentially skewing results toward respondents with positive viewpoints. This can be noted in product reviews or restaurant evaluations.

Several preventative steps can be taken during study design or data collection to mitigate the effects of non-response bias. Ideally, surveys should be as short and accessible as possible, and potential participants should be involved in questions design. Additionally, incentives can be provided for participation if necessary. Lastly, if necessary, surveys can be made mandatory as opposed to voluntary. For example, this could occur if school-age children were initially sent a survey via mail to their homes to complete voluntarily, but this was later changed to a survey required to be completed and handed in at school on an anonymous basis. [38] [39]

Similar to the Hawthorne effect and self-selection bias, recall bias is another potential source of systematic error stemming from the subjects of a particular study. Recall bias is any error due to differences in an individual’s recollections and what truly transpired. Recall bias is particularly prevalent in retrospective studies that use questionnaires, surveys, and/or interviews. [40]

For example, in a retrospective study evaluating the prevalence of cigarette smoking in individuals diagnosed with lung cancer vs. those without, those with lung cancer may be more likely to overestimate their use of tobacco meanwhile those without may underestimate their use. Fortunately, the impact of recall bias can be minimized by decreasing the time interval between an outcome (lung cancer) and exposure (tobacco use). The rationale for this is that individuals are more likely to be accurate when the time period assessed is of shorter duration. Other methods that can be used would be to corroborate the individual’s subjective assessments with medical records or other objective measures whenever possible. [41]

Lastly, in addition to the data collectors and the subjects, bias and subsequent systematic error can be introduced through data analysis, especially if conducted in a manner that gives preference to certain conclusions. There can be blatant data fabrication where non-existing data is reported. However, researchers are more likely to perform multiple tests with pair-wise comparisons, termed “p-hacking.” [42] This typically involves analysis of subgroups or multiple endpoints to obtain statistically significant findings, even if these findings were unrelated to the original hypothesis. P-hacking also occurs when investigators perform data analysis partway through data collection to determine if it is worth continuing or not. [43] It also occurs when covariates are excluded, if outliers are included or dropped without mention, or if treatment groups are split, combined, or otherwise modified based on the original research design. [44] [45]

Ideally, researchers should list all variables explored and all associated findings. If any observations are eliminated (outliers), they should be reported, and an explanation is given as to why they were eliminated and how their elimination affected the data.

Bias in Data Interpretation and Publication

The final stages of any study, interpretation of data and publication of results, is also susceptible to various types of bias. During data interpretation and subsequent discussion, researchers must ensure that the proper statistical tests were used and that they were used correctly. Furthermore, results discussed should be statistically significant, and discussion should be avoided with results that “approach significance.” [46] Furthermore, bias can also be introduced in this stage if researchers discuss statistically significant differences but not clinically significant if conclusions are made about causality when the experiment was purely observational if data is extrapolated beyond the range found within the study. [3]

A major form of bias found during the publication stage is appropriately named publication bias. This refers to the submission of either statistically or clinically significant results, excluding other findings. [47] Journals and publishers themselves have been found to favor studies with significant values. However, researchers themselves may, in turn, use methods of data analysis or interpretation (mentioned above) to uncover significant results. Outcome reporting bias is similar, which refers to the submission of statistically significant results only, excluding non-significant ones. These two biases have been found to affect the results of systematic analyses and even affect the clinical management of patients. [48] However, publication and outcome reporting bias can be prevented in certain cases. Any prospective trials are typically required to be registered before study commencement, meaning that all results, whether significant or not, will be visible. Furthermore, electronic registration and archiving of findings can also help reduce publication bias. [49]

Clinical Significance

Understanding basic aspects of study bias and related concepts will aid clinicians in practicing and improving evidence-based medicine. Study bias can be a major factor that detracts from the external validity of a study or the generalizability of findings to other populations or settings. [50] Clinicians who possess a strong understanding of the various biases that can plague studies will be better able to determine the external validity and, therefore, clinical applicability of a study's findings. [51] [52]

The replicability of a study with similar findings is a strong factor in determining its external validity and generalizability to the clinical setting. Whenever possible, clinicians should arm themselves with the knowledge from multiple studies or systematic reviews on a topic, as opposed to using a single study. [53] Systematic reviews allow applying strategies that limit bias through systematic assembly, appraisal, and unification of the relevant studies regarding a topic. [54]

With a critical, investigational point of view, a willingness to evaluate contrary sources, and the use of systematic reviews, clinicians can better identify sources of bias. In doing so, they can better reduce its impact in their decision-making process and thereby implement a strong form of evidence-based medicine.

Nursing, Allied Health, and Interprofessional Team Interventions

There are numerous sources of bias within the research process, ranging from the design and planning stage, data collection and analysis, interpretation of results, and the publication process. Bias in one or multiple points of this process can skew results and even lead to incorrect conclusions. This, in turn, can cause harmful medical decisions, affecting patients, their families, and the overall healthcare team. Outside of medicine, significant bias can result in erroneous conclusions in academic research, leading to future fruitless studies in the same field. [55]

When combined with the knowledge that most studies are never replicated or verified, this can lead to a deleterious cycle of biased, unverified research leading to more research. This can harm the investigators and institutions partaking in such research and discredit entire fields, even if other investigators had significant work and took extreme care to limit and explain sources of bias.

All research needs to be carried out and reported transparently and honestly. In recent years, important steps have been taken, such as increased awareness of biases present in the research process, manipulating statistics to generate significant results, and implementing a clinical trial registry system. However, all stakeholders of the research process, from investigators to data collectors, to the institutions they are a part of, and the journals that review and publish findings, must take extreme care to identify and limit sources of bias and report those transparently.

All interprofessional healthcare team members, including physicians, physician assistants, nurses, pharmacists, and therapists, need to understand the variety of biases present throughout the research process. Such knowledge will separate stronger studies from weaker ones, determine the clinical and real-world applicability of results, and optimize patient care through the appropriate use of data-driven research results considering potential biases. Failure to understand various biases and how they can skew research results can lead to suboptimal and potentially deleterious decision-making and negatively impact both patient and system outcomes.

Review Questions
Access free multiple choice questions on this topic.
Comment on this article.

Disclosure: Aleksandar Popovic declares no relevant financial relationships with ineligible companies.

Disclosure: Martin Huecker declares no relevant financial relationships with ineligible companies.

This book is distributed under the terms of the Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International (CC BY-NC-ND 4.0) ( http://creativecommons.org/licenses/by-nc-nd/4.0/ ), which permits others to distribute the work, provided that the article is not altered or used commercially. You are not required to obtain permission to distribute this article, provided that you credit the author and journal.

Cite this Page Popovic A, Huecker MR. Study Bias. [Updated 2023 Jun 20]. In: StatPearls [Internet]. Treasure Island (FL): StatPearls Publishing; 2024 Jan-.

In this Page

Bulk download.

Bulk download StatPearls data from FTP

Related information

PMC PubMed Central citations
PubMed Links to PubMed

Recent Activity

Study Bias - StatPearls Study Bias - StatPearls

Your browsing activity is empty.

Activity recording is turned off.

Turn recording back on

Connect with NLM

National Library of Medicine 8600 Rockville Pike Bethesda, MD 20894

Web Policies FOIA HHS Vulnerability Disclosure

Help Accessibility Careers

Log in using your username and password

Search More Search for this keyword Advanced search
Latest content
Current issue
Write for Us
BMJ Journals More You are viewing from: Google Indexer

Joanna Smith 1 ,
Helen Noble 2
1 School of Human and Health Sciences, University of Huddersfield , Huddersfield , UK
2 School of Nursing and Midwifery, Queens's University Belfast , Belfast , UK
Correspondence to : Dr Joanna Smith , School of Human and Health Sciences, University of Huddersfield, Huddersfield HD1 3DH, UK; j.e.smith{at}hud.ac.uk

https://doi.org/10.1136/eb-2014-101946

Statistics from Altmetric.com

Request permissions.

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

The aim of this article is to outline types of ‘bias’ across research designs, and consider strategies to minimise bias. Evidence-based nursing, defined as the “process by which evidence, nursing theory, and clinical expertise are critically evaluated and considered, in conjunction with patient involvement, to provide the delivery of optimum nursing care,” 1 is central to the continued development of the nursing professional. Implementing evidence into practice requires nurses to critically evaluate research, in particular assessing the rigour in which methods were undertaken and factors that may have biased findings.

What is bias in relation to research and why is understanding bias important?

Although different study designs have specific methodological challenges and constraints, bias can occur at each stage of the research process ( table 1 ). In quantitative research, the validity and reliability are assessed using statistical tests that estimate the size of error in samples and calculating the significance of findings (typically p values or CIs). The tests and measures used to establish the validity and reliability of quantitative research cannot be applied to qualitative research. However, in the broadest context, these terms are applicable, with validity referring to the integrity and application of the methods and the precision in which the findings accurately reflect the data, and reliability referring to the consistency within the analytical processes. 4

View inline

Types of research bias

How is bias minimised when undertaken research?

Bias exists in all study designs, and although researchers should attempt to minimise bias, outlining potential sources of bias enables greater critical evaluation of the research findings and conclusions. Researchers bring to each study their experiences, ideas, prejudices and personal philosophies, which if accounted for in advance of the study, enhance the transparency of possible research bias. Clearly articulating the rationale for and choosing an appropriate research design to meet the study aims can reduce common pitfalls in relation to bias. Ethics committees have an important role in considering whether the research design and methodological approaches are biased, and suitable to address the problem being explored. Feedback from peers, funding bodies and ethics committees is an essential part of designing research studies, and often provides valuable practical guidance in developing robust research.

In quantitative studies, selection bias is often reduced by the random selection of participants, and in the case of clinical trials randomisation of participants into comparison groups. However, not accounting for participants who withdraw from the study or are lost to follow-up can result in sample bias or change the characteristics of participants in comparison groups. 7 In qualitative research, purposeful sampling has advantages when compared with convenience sampling in that bias is reduced because the sample is constantly refined to meet the study aims. Premature closure of the selection of participants before analysis is complete can threaten the validity of a qualitative study. This can be overcome by continuing to recruit new participants into the study during data analysis until no new information emerges, known as data saturation. 8

In quantitative studies having a well-designed research protocol explicitly outlining data collection and analysis can assist in reducing bias. Feasibility studies are often undertaken to refine protocols and procedures. Bias can be reduced by maximising follow-up and where appropriate in randomised control trials analysis should be based on the intention-to-treat principle, a strategy that assesses clinical effectiveness because not everyone complies with treatment and the treatment people receive may be changed according to how they respond. Qualitative research has been criticised for lacking transparency in relation to the analytical processes employed. 4 Qualitative researchers must demonstrate rigour, associated with openness, relevance to practice and congruence of the methodological approach. Although other researchers may interpret the data differently, appreciating and understanding how the themes were developed is an essential part of demonstrating the robustness of the findings. Reducing bias can include respondent validation, constant comparisons across participant accounts, representing deviant cases and outliers, prolonged involvement or persistent observation of participants, independent analysis of the data by other researchers and triangulation. 4

In summary, minimising bias is a key consideration when designing and undertaking research. Researchers have an ethical duty to outline the limitations of studies and account for potential sources of bias. This will enable health professionals and policymakers to evaluate and scrutinise study findings, and consider these when applying findings to practice or policy.

Wakefield AJ ,
Anthony A ,
↵ The Lancet . Retraction—ileal-lymphoid-nodular hyperplasia, non-specific colitis, and pervasive developmental disorder in children . Lancet 2010 ; 375 : 445 . OpenUrl CrossRef PubMed Web of Science
Easterbrook PJ ,
Berlin JA ,
Gopalan R ,
Petticrew M ,
Thomson H ,
Francis J ,
Johnston M ,
Robertson C ,

Competing interests None.

Read the full text or download the PDF:

Research article
Open access
Published: 23 March 2017

The impact of non-response bias due to sampling in public health studies: A comparison of voluntary versus mandatory recruitment in a Dutch national survey on adolescent health

Kei Long Cheung ORCID: orcid.org/0000-0001-7648-4556 1 ,
Peter M. ten Klooster 2 ,
Cees Smit 3 ,
Hein de Vries 4 &
Marcel E. Pieterse 5

BMC Public Health volume 17 , Article number: 276 ( 2017 ) Cite this article

44k Accesses

235 Citations

32 Altmetric

Metrics details

In public health monitoring of young people it is critical to understand the effects of selective non-response, in particular when a controversial topic is involved like substance abuse or sexual behaviour. Research that is dependent upon voluntary subject participation is particularly vulnerable to sampling bias. As respondents whose participation is hardest to elicit on a voluntary basis are also more likely to report risk behaviour, this potentially leads to underestimation of risk factor prevalence. Inviting adolescents to participate in a home-sent postal survey is a typical voluntary recruitment strategy with high non-response, as opposed to mandatory participation during school time. This study examines the extent to which prevalence estimates of adolescent health-related characteristics are biased due to different sampling methods, and whether this also biases within-subject analyses.

Cross-sectional datasets collected in 2011 in Twente and IJsselland, two similar and adjacent regions in the Netherlands, were used. In total, 9360 youngsters in a mandatory sample (Twente) and 1952 youngsters in a voluntary sample (IJsselland) participated in the study. To test whether the samples differed on health-related variables, we conducted both univariate and multivariable logistic regression analyses controlling for any demographic difference between the samples. Additional multivariable logistic regressions were conducted to examine moderating effects of sampling method on associations between health-related variables.

As expected, females, older individuals, as well as individuals with higher education levels, were over-represented in the voluntary sample, compared to the mandatory sample. Respondents in the voluntary sample tended to smoke less, consume less alcohol (ever, lifetime, and past four weeks), have better mental health, have better subjective health status, have more positive school experiences and have less sexual intercourse than respondents in the mandatory sample. No moderating effects were found for sampling method on associations between variables.

Conclusions

This is one of first studies to provide strong evidence that voluntary recruitment may lead to a strong non-response bias in health-related prevalence estimates in adolescents, as compared to mandatory recruitment. The resulting underestimation in prevalence of health behaviours and well-being measures appeared large, up to a four-fold lower proportion for self-reported alcohol consumption. Correlations between variables, though, appeared to be insensitive to sampling bias.

Peer Review reports

When monitoring health indicators and risk behaviour among adolescent populations, it is important to understand the magnitude of selective non-response and the impact this may have on the prevalence estimates. As described by Berg [ 1 ]: “non-response bias refers to the mistake one expects to make in estimating a population characteristic based on a sample of survey data in which, due to non-response, certain types of survey respondents are under-represented” (p. 3). It seems that non-response bias is the rule rather than the exception in epidemiological surveys, and this is long recognised [ 2 ]. Literature on non-response bias through mailed surveys shows that non-response bias is a serious concern in survey studies [ 3 , 4 ].

Selective non-response may be associated with general characteristics of the study population. Previous studies have shown that female, older individuals, and individuals with higher education levels are more prone to return postal questionnaires [ 5 , 6 ]. In such cases biased prevalence estimates are often corrected by controlling for these demographic variables or by estimating weighted proportions [ 7 ]. However, selective non-response may also be due to the actual outcome variables of interest. Studies generally show that respondents in health surveys report better health status and more positive health-related behaviours than non-respondents, including self-rated health and chronic diseases, smoking, physical inactivity, obesity, [ 5 , 8 , 9 ], lower alcohol consumption [ 10 – 12 ], better mental health, better subjective health status, more positive school experiences [ 13 – 25 ], and less risky sexual behaviour [ 16 ] than non-respondents. These findings indicate that people with poorer health tend to avoid participating in health surveys. While there are many factors that are important in ensuring the generalisability of findings in health studies, unbiased subject sampling may be paramount. Due to subject self-selection, research that is dependent upon voluntary subject participation is particularly vulnerable to sampling bias [ 26 ]. Respondents whose participation is hardest to elicit are likely to report more risk behaviour [ 27 , 28 ]. In spite of this, the literature on the methodological implications of non-response due to sampling methods seems rather limited, and pertaining to adolescent populations in particular [ 17 , 18 , 24 , 27 ]. Therefore, this study investigates the impact of non-response bias on prevalence estimates among adolescents, by comparing data gathered through voluntary sampling (with a high non-response rate) with data gathered through a mandatory sampling strategy (with a high participation rate).

As the validity of prevalence estimates within a population may be affected by non-response, this may also apply to analyses of between-variable associations within such datasets. For example, adolescent research has shown that various health risks appear to cluster in individuals [ 29 , 30 ], presumably the result of shared underlying distal determinants like low self-esteem [ 31 ] or adverse personality traits [ 32 ]. Therefore, when studying the causal mechanisms underlying adolescent health risk behaviour by analysing co-variates of these behaviours, it is conceivable that these analyses may be confounded by selective non-response [ 8 ]. In other words, it seems warranted to investigate whether non-response bias may, indirectly, moderate associations among health-related variables. Although a non-response bias in itself cannot be a true moderating variable, it may be considered as a latent moderator that represents effects of true moderators that in turn are affected by non-response. Examples of such moderators within the field of substance use research are demographic characteristics. Studies indicate that demographics may moderate associations between tobacco consumption on the one hand, and for example alcohol consumption, school experiences, mental health, subjective health status on the other. Similarly, associations between alcohol consumption and school experiences, mental health, and subjective health status may be affected by demographic variables [ 33 – 40 ]. For example, gender differences were found in patterns of association between substance use and mood disorders [ 33 ], and for the association between tobacco consumption and drinking [ 36 , 41 ]. Summarizing, this evidence implies that a non-response bias may affect the demographic composition of a sample [ 5 , 6 ], and these demographics in turn are known to moderate associations between other health-related variables. Similarly, this may also apply to other mechanisms through which a non-response bias may invalidate between-variable associations in epidemiological research.

In order to enhance our understanding of non-response bias in public health monitoring of adolescents, this study first aims to identify whether there are systematic differences in prevalence estimates between two similar samples but with different rates of non-response due to sampling strategy. Biases in prevalence estimates are tested for both demographic and health characteristics, in two ways: by comparing the observed rates in both samples with the estimates known from available population statistics, and by testing the differences between both samples directly. Second, as it is conceivable that due to a non-response bias associations between risk factors will be confounded, this study also examines sampling method (mandatory recruiting with high response rate vs. voluntary recruiting with low response rate) as a latent moderator of associations between health related variables within subjects.

Sampling methods

Seven Community Health Services [CHSs] in the eastern part of the Netherlands collaborated with Maastricht University on the project named E-MOVO, a Dutch acronym for Electronic Monitor and Health Education [ 42 ]. E-MOVO is an electronic monitoring instrument, aimed at providing insight into health of adolescents of the 8th and 10th graders of secondary education. Whereas in most regions participation for adolescents at participating schools was mandatory, regions had the option to choose another sampling method. We used the results of two regions which used two different ways of sampling. In the mandatory sample (region Twente) sampling occurred mandatory and adolescents were recruited via secondary schools. Students in participating schools were instructed to complete the online questionnaire during a single class session (approximately 45 min) [ 43 ]. In the voluntary sample (region IJsseland) the adolescents were recruited voluntarily and were invited via a postal mailing to their home address, containing a hyperlink and personal code to the online questionnaire.

Non-response bias in the mandatory sample is considered minimal, as non-participation occurs in clusters (i.e. schools and classes) instead of the individual level. Each school in the region was invited to have all classes participate. There were several schools that did not participate at all, and some participating schools did not include all classes, due to practical reasons such as scheduling difficulties and lack of computer rooms. Therefore, we assume that there is minimal non-response bias in the data of the mandatory sample at the individual level. In contrast, due to higher non-response in the voluntary sample, it is likely that there is more non-response bias compared to the mandatory sample, as non-respondents here may differ in several characteristics from respondents.

An important requirement for the purpose of this study is that both populations from which the two samples were recruited are indeed comparable. Both regions are geographically adjacent, and similar with respect to socio-economic and urbanisation characteristics. With regard to risk behaviour prevalence, interregional comparability can be verified with two Dutch data resources on alcohol and tobacco consumption. In both resources data were collected across all regions with a standardized recruitment strategy and questionnaire, allowing direct interregional comparisons without a differential bias due to non-response. First, in the Health Monitor of 2012, with a representative sample of Dutch adults of 19 years and older, smoking prevalence was estimated at 23.9% in Twente and 22.0% in IJsselland [ 44 , 45 ]. Weekly prevalence of heavy drinking (consuming 5 or more standard units on a single day at least once a week) was estimated at 9.2% in Twente and 8.7% in IJsselland [ 44 , 45 ]. Second, the Dutch Health Survey with a representative sample of Dutch individuals of 12 years and older, identified the percentage smokers in 2008 at 32.3% in Twente and 29.8% in IJsselland. Hazardous drinking prevalence, defined in this study as either heavy drinking or exceeding moderate drinking levels (≥14 units a week for females and 21 units for males), was estimated in 2008 at 20.7% in Twente and 19.8% in IJsselland [ 46 ]. In general, available national data show that both regions included in this study show negligible differences in alcohol consumption, and a small difference in smoking prevalence. Although these data could not be specified for adolescents, in the case of the Dutch Health Survey adolescents of 12 year and older were included in the estimates. Nevertheless, it seems reasonable to assume that the magnitude of interregional differences found among adults may also apply to the adolescent populations of these regions.

Participants

In the mandatory sample, the CHS of Twente was involved in recruiting schools in the 2011 study and maintained contact with its 14 municipalities within the region. All 59 secondary schools were approached, from which 39 participated in the E-MOVO study of 2011. The research team of E-MOVO informed the municipalities via e-mail about the study. The CHS of Twente informed each municipality and recruited schools within the community by sending an information sheet. Within participating schools informed consent was obtained from parents via an opt-out procedure. In the voluntary sample, the CHS of IJsselland selected a random sample of youngsters between the ages of 12 and 23, stratified on all municipalities in the region. For comparison of the regions, only the ages from 13 through 16 were included. Informed consent was obtained by sending a postal mail to the parents with an information sheet and the invitation for their child to participate.

All matching items between the two surveys (Twente and IJsseland) were analysed. Measures were based on self-reports which have been shown to be reliable regarding tobacco, alcohol, and other drug use among adolescents [ 47 , 48 ].

Demographics

Gender, age (in years), and education (11 options in Twente, 15 options in IJsselland) were assessed. For analytic purposes, education was dichotomised into low (“preparatory middle-level vocational education”) or high (“higher general continued education”/“preparatory scholarly education”).

Tobacco consumption

Participants were asked how often they smoked at present (0 = not at all; 1 = less than once a week; 2 = at least once a week, 3 = but not daily; 4 = every day). As previous studies reported whether or not youngsters smoke daily and due to violation of the linearity assumption, tobacco consumption was dichotomised into ‘daily smoker’ and ‘non-daily smoker’ [ 49 , 50 ].

Alcohol consumption

Alcohol consumption was operationalised with three items. Participants were asked whether they had ever consumed alcohol (yes; no), how often they had had alcohol in their lives, and how often they had consumed alcohol in the past four weeks (0; 1; 2; 3; 4; 5; 6; 7; 8; 9; 10; 11–19; >20 times). As multiple reports mention whether youngsters had or had not consumed alcohol in the past four weeks [ 49 , 50 ] and due to violation of the linearity assumption, alcohol in the past four weeks was dichotomised (yes/no).

Mental health

The Strengths and Difficulties Questionnaire [SDQ] is a behavioural screening questionnaire for children aged 4–16 years [ 51 , 52 ]. The SDQ consists of 25 items and measures five scales of five items each (i.e. emotional symptoms, conduct problems, hyperactivity-inattention, peer problems, and prosocial behaviour). It has been extensively validated in many countries [ 53 , 54 ]. The internal consistency (Cronbach’s alpha of .64), test-retest stability (except for the prosocial behaviour subscale (.59), all intraclass correlation coefficients were above.70), and parent-youth agreement of the various SDQ scales have been found acceptable [ 54 ]. To estimate the ‘probability for any behavioural problems from the SDQ scores, a modified version of Goodman’s algorithm [ 51 ] was used for the total score. Based on the algorithm, the probability of a psychiatric disorder was calculated as ‘1 = unlikely’ (0–15), ‘2 = possible’ (16–19), and ‘3 = probable’ (20–40) [ 51 ].

Subjective health status

One item was used to measure the subjective health status, consistent with other studies (e.g. DeSalvo, Bloser, Reynolds, He, & Muntner, 2006 [ 55 ]) Individuals were asked how they perceived their health in general (1 = very good; 2 = good; 3 = neutral; 4 = not good; 5 = poor).

School experiences

Participants were asked with one item how they experienced school (1 = great fun; 2 = fun; 3 = neutral; 4 = not fun; 5 = dreadful).

Sexual behaviour

In order to measure sexual behaviour one item was used [ 56 ]. Individuals were asked whether they had ever had sexual intercourse with someone (1 = never; 2 = once; 3 = couple of times; 5 = regularly).

Statistical analyses

First, for both samples we examined whether the observed distribution of demographics deviated from the expected distribution in the population. For gender, a one sample t-test was performed. For the distribution of age and education level we provided descriptive comparisons of the mean age and education level (high vs. low) of the samples to the population estimates available to the best of our knowledge. Statistical tests were not performed with these demographic variables as the reliability of these estimates was lower than for gender.

Second, tests were performed of differences between both samples. For demographic characteristics, an independent samples t -test was used for age, and Pearson χ 2 -test for gender and education level (high vs low). To examine whether the samples differed on health-related variables, we first conducted univariable logistic regression analyses for each health-related variable of interest as independent variable and sampling method as dependent variable (mandatory sample Twente =0, voluntary sample IJsselland =1). Although, theoretically, sampling method would be considered as the independent variable, this was reversed in these analyses to allow a uniform analysis technique to be used for all health-related variables, regardless of the different measurement levels of these variables.

For the logistic regression analyses we checked the linearity assumption for non-binary variables (i.e. sexual intercourse, subjective health, school experiences, tobacco consumption, alcohol in past four weeks, lifetime alcohol consumption, and SDQ). Except for SDQ, alcohol in past four weeks, and tobacco consumption, variables did not violate the linearity assumption. To solve this issue, these three outcome measures were recoded into binary (tobacco consumption: 0 = no daily smoker, 1 = daily smoker; alcohol past four weeks: 0 = no, 1 = yes) or three-level (SDQ: 1 = unlikely, 2 = possible, 3 = likely). Further, to examine whether the differences in health characteristics between the samples could be explained by differences in demographic characteristics, all multivariable logistic regression analyses were repeated, with demographics (i.e. age, gender, and education) added as covariates. Intercorrelations were checked to test for collinearity between the health-related variable and demographic variables entered into the model. No signs of collinearity issues were found among the independent variables with all tolerance levels above 0.1 [ 57 ] and VIF values below 10 [ 58 ].

To examine moderation effects of sampling bias on associations between health related variables within subjects, an interaction term was computed for sampling method with tobacco consumption. Then interaction analyses were performed using logistic regression analysis according to the procedure by Baron and Kenny [ 59 ], with tobacco consumption, sampling method, and the sampling*tobacco use interaction term entered as independent variables. As independent variables the following health variables were tested in consecutive models: mental health, subjective health status, and school experiences. The same procedure was followed for tobacco consumption, alcohol consumption, and alcohol in past four weeks as dichotomous dependent variables. Due to the large sample size in this study a significance level of <0.01 was used in all analyses. All analyses were carried out using SPSS 20.0.

Sample characteristics

A total of 9360 8th and 10th graders (49.2% female) of secondary education were enrolled in in the mandatory sample. In the voluntary sample, a total of 1952 youngsters (55.8% female) participated. All sample characteristics are depicted in Table 1 .

Comparing demographic characteristics of both samples with population estimates

Findings supported the assumption that voluntary recruiting leads to more selective non-response than mandatory recruiting. A one sample t-test showed that the distribution of gender in the voluntary sample (55.8% female) deviated considerably from available population estimates (48.5% female), received from the CHS of IJsselland, t (1951) = 6.038, ( p < 0.01). Using the population estimates from the CHS of Twente, no significant deviation was found in the mandatory sample regarding gender.

In addition, the estimated mean age for the population of interest in IJsselland [ 60 ] was slightly higher (14.5 y) than the age observed in the voluntary sample (14.3 y). Almost no difference was observed in Twente, with an average age of 14.2 years in the estimated population and 14.1 years in the mandatory sample. For education, the discrepancy between the expected proportion of highly educated (HAVO/VWO) students (50.0%) [ 49 ] and the observed rates was substantially higher in IJsselland (61.3%) than in Twente (51.5%). Overall, compared to the voluntary sample, the mandatory sample appeared less affected by non-response bias with respect to demographics.

Effects of sampling bias on demographic characteristics

The average age of participants in the voluntary sample ( M = 14.29, SD = 1.07, N = 1571) was not significantly different at the predefined 0.01 level from participants in the mandatory sample ( M = 14.23, SD = 1.14, N = 8761; t (10,330) = 2.03, p = 0.04, two-tailed). The percentage of females in the voluntary sample’s (55.8%) was higher compared to the mandatory sample’s (49.2%; χ 2 (1) = 28.380, p < 0.01). For education, the percentage of high education students in the voluntary sample (61.3%) was higher than in the mandatory sample (51.5%; χ 2 (1) = 55.91, p < 0.01).

Effects of sampling bias on health related variables

Bivariate analyses of health related measures revealed several differences between the mandatory sample and the voluntary sample (Table 2 ). Individuals in the mandatory sample reported worse school experiences (OR = 0.54; 95% CI = 0.50–0.58) and subjective health (OR = 0.80; 95% CI = 0.74–0.86) than individuals in the voluntary sample. Based on the SDQ, a higher prevalence of individuals with a ‘possible’ psychiatric disorder was observed in the mandatory sample (OR = 0.67; 95% CI = 0.54–0.83). No difference was found in the prevalence of a ‘probable’ psychiatric disorder. More participants in the mandatory sample than in the voluntary sample reported daily smoking (OR = 0.37; 95% CI = 0.28–0.47) and having sexual intercourse (OR = 0.71; 95% CI = 0.64–0.78). Regarding alcohol consumption, bivariate odds ratios indicated that more individuals in the mandatory sample had ever consumed alcohol (OR = 0.33; 95% CI = 0.30–0.37). Respondents in the mandatory sample also reported more lifetime alcohol consumption (OR = 0.84; 95% CI = 0.83–0.85) and more recent alcohol use (in the past four weeks) (OR = 0.18; 95% CI = 0.15–0.21). When adjusting for gender, education, and age in multivariable regression analyses (see Table 2 ) similar odds ratios were found on all health related variables, with 95% confidence intervals largely overlapping in all cases. This indicates that despite controlling for demographic differences, lower tobacco consumption, lower alcohol consumption, better mental health, better subjective health status, more positive school experiences, and less sexual behaviour were found in the voluntary sample compared to the mandatory sample.

Effects of sampling bias on within-subject analyses

Remarkably, no support was found for a moderating role of sampling method on any of the associations between tobacco consumption and one of the following: alcohol consumption, school experiences, mental health, and subjective health status. Similarly, no moderation effects of sampling were found on associations between alcohol consumption and any other health related variables.

The primary aim of this study was to investigate potential effects of non-response bias on prevalence estimates of self-reported health behaviours and well-being, comparing samples obtained from a similar population but with different recruitment strategies and with different non-response ratios. Results showed strong and consistent effects of non-response on all health estimates, as well as considerable effects on the distribution of demographic characteristics. As expected, non-response unambiguously contributed to underestimated health risks.

Expectations derived from literature [ 6 , 15 ] concerning demographic differences between non-respondents and respondents were confirmed in this study. We found that female, older individuals, and persons with higher education were over-represented in the voluntary sample, while the mandatory sample approached the norm population on these variables. Thus, different sampling methods may recruit different participants, and these demographic differences may be fairly substantial.

Bias due to selective non-response also occurred in health related variables. In line with previous studies [ 7 , 11 – 27 ], we found that voluntary respondents report more favourable health indicators, e.g. less smoking, less alcohol consumption, better mental health, better subjective health, more positive school experiences, and less risky sexual behaviour than mandatory respondents. Overall, observed differences between the two samples appeared large to very large, in particular concerning school experiences and alcohol consumption in the past four weeks. For instance, the proportion of respondents who reported alcohol use in the past four weeks was four times higher in the mandatory sample than in the voluntary sample. This is even higher than reported in previous studies regarding alcohol consumption of adults (in which non-response bias was assessed by comparing early and late responders as proxies [ 61 , 62 ]. Thus, this study indicates that voluntary recruitment may lead to severe underestimation of health-related risk behaviour and mental health problems, compared to mandatory recruitment. Interestingly, this underestimation effect remained highly significant after controlling for the demographic variables. Perhaps being confronted with one’s harmful (smoking), illicit (underage alcohol use), or intimate (sexual behaviour) practices by filling in a survey is perceived as unpleasant (or too private) and motivates these individuals to withdraw from partaking in the survey [ 8 ]. These results corroborate recent literature indicating that surveys underestimate risk behaviour due to selective non-response and that this bias increases as response rates fall [ 28 ]. Moreover, this study adds that when a controversial topic is involved, motives for not participating are predominantly related to the topic itself, rather than to more generic characteristics [ 5 ]. This also implies that in such cases calculating weighted estimates of health related risks to correct for underrepresentation of demographic characteristics would not be sufficient.

Finally, no differences were found between the samples in the strength of the associations between tobacco consumption and alcohol consumption on the one hand and other risk factors on the other. This indicates that non-response did not confound any of these examined associations. Apparently, non-response bias does affect prevalence estimates but within-subject analyses are rather insensitive to such a bias. This may imply that the non-responding boys (or smokers, drinkers, etc.) do not deviate from their responding peers with respect to mechanisms underlying these health-related behaviours in a systematic way. These groups are primarily underrepresented in numbers due to a reluctance to reveal socially undesirable habits. This may have important implications in particular for research on causal mechanisms underlying harmful behaviour and decreased mental health, as a high non-response rate not necessarily poses a threat to the validity of such studies [ 8 ].

Clearly, the interpretation of the effect sizes found in this study should be taken with caution as these may depend on the specific characteristics of the samples included in this particular study. Regardless whether these represent small or large effects, however, even small effects may have large impact in public health research. It is argued that the translation of effect size estimates to the assessment of practical importance is not straightforward. Many considerations of the context (e.g. measurement, methodology, and empirical evidence) should be factored into assessments of practical importance [ 63 ]. Numerous studies in psychology address important psychological variables or processes, despite the fact that many of them have yielded small effects [ 64 ]. Within the context of public health, even small effects in estimates due to non-response bias are relevant.

Limitations

This study is not without limitations. Two of our central assumptions may not hold, i.e. that (1) there is minimal non-response bias in the mandatory sample and (2) that the true populations in Twente and IJsselland are not intrinsically different. With regard to the first assumption, there is a possibility that youngsters from participating schools in the mandatory sample may not be generalizable to the population of Twente, in spite of the negligible deviations found on demographic characteristics in comparison with population estimates from Twente. The population estimates available may have been insufficient to rigorously test this assumption, as these data have not been published under peer review. The same holds for the IJsselland region. Moreover, in the mandatory sample non-participating schools may differ from participating schools in characteristics relevant to the topic of this study. For instance, non-participating schools may be more likely to be located in deprived neighbourhoods. However, such a bias in the mandatory sample would be likely to contribute to an underestimation of the prevalence of most risk factors within the mandatory sample. This would imply that the true contrast in prevalence estimates between the mandatory and voluntary sample would be even more pronounced than within our current data.

The second assumption, that the adolescent populations in Twente and IJsselland are comparable (as the regions are adjacent, part of the same province, and share similar cultural and topographic characteristics), was partially verified. Two national data sets show a slightly higher smoking prevalence in Twente, and a negligible difference in alcohol consumption among the total population [ 44 – 46 ]. Some caution is needed, as we extrapolated the regional comparisons among the adult population to adolescents. Yet, even when taking this into account, the observed differences in alcohol use and smoking prevalence by far exceed any differences found in both national data sets. For example, the difference in smoking prevalence from the 2012 Health Monitor (23.9% vs. 22.0%) amounts to a relative risk of 1.08, whereas the difference observed between our adolescent samples (9.1% vs 3.5%) equals a relative risk of 2.60. And in the case of alcohol use this contrast is even more distinct. Moreover, the effect sizes found on health related variables remained mostly unchanged when controlling for demographic differences. Therefore, it seems justified to conclude that the consistent underestimation in risk estimates found in this study resulted primarily from non-response bias, and that confounding by true regional differences can only be very small to almost negligible.

Future research may investigate whether our results are replicable in a more controlled design, comparing mandatory and voluntary sampling from an identical population.

This study is to our knowledge the first to provide direct evidence that the extent of non-response bias in health studies depends on sampling method. Using an identical online survey, a dataset obtained through a mandatory sampling method (school-based) with minimal non-response, was compared to data collected with the more common voluntary sampling method (postal invitation) with presumably a much higher non-response rate. Fortunately, the difference in sampling method did not seem to bias the associations between health-related variables. This suggests that for correlational and longitudinal cohort studies examining within-subject associations between risk factors and health behaviour, non-response bias is not likely to threaten the validity of the results. However, the prevalence of self-reported health variables – tobacco consumption, alcohol consumption, mental health, subjective health status, school experiences, and sexual behaviour - may be substantially underestimated due to selective non-response effects. The large effect sizes we found may have implications for researchers and health policy makers. Researchers should be cautious when recruiting participants for health studies with voluntary recruiting, in particular among adolescents. When the aim is to estimate prevalence or monitor changes over time in prevalence, trends may be missed or mistakenly observed due to non-response bias. And when using voluntary sampling, researchers should employ methods to maximise response rates, and consider data analysis techniques to account for a non-response bias as much as possible. Policy makers should be aware of the likelihood of underestimating adolescent health risks when based on surveys with low response rates.

Abbreviations

Community health service

Electronic monitor and health education

Strengths and difficulties questionnaire

Variance inflation factor

Berg N. Non-response Bias. Encyclopedia of Social Measurement 2: 865–873. Kempf-Leonard, K., ed. London: Academic Press; 2010.

Google Scholar

Locker D, Wiggins R, Sittampalam Y, et al. Estimating the prevalence of disability in the community: the influence of sample design and response bias. J Epidemiol Community Health. 1981;35(3):208–12.

Article CAS PubMed PubMed Central Google Scholar

Dillman DA. Mail and telephone surveys: Wiley Interscience; 1978.

Groves RM, Cialdini RB, Couper MP. Understanding the decision to participate in a survey. Public Opin Quarterly. 1992;56(4):475–95.

Article Google Scholar

Criqui MH, Barrett-Connor E, Austin M. Differences between respondents and non-respondents in a population-based cardiovascular disease study. Am J Epidemiol. 1978;108(5):367–72.

Article CAS PubMed Google Scholar

Jooste P, Yach D, Steenkamp H, et al. Drop-out and newcomer bias in a community cardiovascular follow-up study. Int J Epidemiol. 1990;19(2):284–9.

Little RJ, Vartivarian S. On weighting the rates in non-response weights. Stat Med. 2003;22(9):1589–99.

Article PubMed Google Scholar

Van Loon AJM, Tijhuis M, Picavet HSJ, et al. Survey non-response in the Netherlands: effects on prevalence estimates and associations. Ann Epidemiol. 2003;13(2):105–10. PubMed PMID: WOS:000180845000004. English

Paganini-Hill A, Hsu G, Chao A, et al. Comparison of early and late respondents to a postal health survey questionnaire. Epidemiology. 1993;4(4):375–9.

Lemmens P, Tan E, Knibbe R. Bias due to non-response in a Dutch Survey on Alcohol Consumption. Br J Addict. 1988;83(9):1069–77.

Pernanen K. Validity of survey data on alcohol use. Res Adv Alcohol Drug Probl. 1974;1:355–74.

Wild TC, Cunningham J, Adlaf E. Nonresponse in a follow-up to a representative telephone survey of adult drinkers. J Stud Alcohol Drugs. 2001;62(2):257.

Article CAS Google Scholar

Boeing H, Korfmann A, Bergmann M. Recruitment procedures of EPIC-Germany. Ann Nutr Metab. 1999;43(4):205–15.

Boström G, Hallqvist J, Haglund BJ, et al. Socioeconomic differences in smoking in an Urban Swedish population the bias introduced by non-participation in a mailed questionnaire. Scand J Public Health. 1993;21(2):77–82.

Criqui MH. Response bias and risk ratios in epidemiologic studies. Am J Epidemiol. 1979;109(4):394–9.

Dunne MP, Martin NG, Bailey JM, et al. Participation bias in a sexuality survey: psychological and behavioural characteristics of responders and non-responders. Int J Epidemiol. 1997;26(4):844–54.

Frame CL, Strauss CC. Parental informed consent and sample bias in grade-school children. J Soc Clin Psychol. 1987;5(2):227–36.

Gerrits MH, Voogt R, van den Oord EJ. An evaluation of nonresponse bias in peer, self, and teacher ratings of children's psychosocial adjustment. J Child Psychol Psychiatry. 2001;42(05):593–602.

Hill A, Roberts J, Ewings P, et al. Non-response bias in a lifestyle survey. J Public Health. 1997;19(2):203–7.

Jacobsen BK, Thelle DS. The Tromsø Heart Study: responders and non-responders to a health questionnaire, do they differ? Scand J Public Health. 1988;16(2):101–4.

CAS Google Scholar

Janzon L, Hanson BS, Isacsson S-O, et al. Factors influencing participation in health surveys. Results from prospective population study'Men born in 1914'in Malmö, Sweden. J Epidemiol Community Health. 1986;40(2):174–7.

Kessler RC, Little RJ, Groves RM. Advances in strategies for minimizing and adjusting for survey nonresponse. Epidemiol Rev. 1995;17(1):192–204.

Macera CA, Jackson KL, Davis DR, et al. Patterns of non-response to a mail survey. J Clin Epidemiol. 1990;43(12):1427–30.

Noll RB, Zeller MH, Vannatta K, et al. Potential bias in classroom research: comparison of children with permission and those who do not receive permission to participate. J Clin Child Psychol. 1997;26(1):36–42.

O'neill T, Marsden D, Silman A. Differences in the characteristics of responders and non-responders in a prevalence survey of vertebral osteoporosis. Osteoporos Int. 1995;5(5):327–34.

Rosenthal R, Rosnow RL. The volunteer subject. 1975.

Kypri K, Samaranayaka A, Connor J, et al. Non-response bias in a web-based health behaviour survey of New Zealand tertiary students. Prev Med. 2011;53(4):274–7.

Maclennan B, Kypri K, Langley J, et al. Non-response bias in a community survey of drinking, alcohol-related experiences and public opinion on alcohol policy. Drug Alcohol Depend. 2012;126(1):189–94.

Ary DV, Duncan TE, Duncan SC, et al. Adolescent problem behavior: the influence of parents and peers. Behav Res Ther. 1999;37(3):217–30.

Tyas SL, Pederson LL. Psychosocial factors related to adolescent smoking: a critical review of the literature. Tob Control. 1998;7(4):409–20.

Baumeister RF, Campbell JD, Krueger JI, et al. Does high self-esteem cause better performance, interpersonal success, happiness, or healthier lifestyles? Psychol Sci Public Interest. 2003;4(1):1–44.

J Conrod P, Nikolaou K. Annual research review: on the developmental neuropsychology of substance use disorders. J Child Psychol Psychiatry. 2016;57(3):371–94.

Alati R, Kinner S, Najman JM, et al. Gender differences in the relationships between alcohol, tobacco and mental health in patients attending an emergency department. Alcohol Alcohol. 2004;39(5):463–9.

Caldwell TM, Rodgers B, Jorm AF, et al. Patterns of association between alcohol consumption and symptoms of depression and anxiety in young adults. Addiction. 2002;97(5):583–94.

Covey LS, Hughes DC, Glassman AH, et al. Ever-smoking, quitting, and psychiatric disorders: evidence from the Durham, North Carolina, Epidemiologic Catchment Area. Tobacco Control. 1994;3(3):222.

Article PubMed Central Google Scholar

Craig TJ, Natta PAV. The association of smoking and drinking habits in a community sample. J Stud Alcohol Drugs. 1977;38(07):1434.

Degenhardt L, Hall W. The relationship between tobacco use, substance-use disorders and mental health: results from the National Survey of Mental Health and Well-being. Nicotine Tob Res. 2001;3(3):225–34.

Degenhardt L, Hall W. Patterns of co-morbidity between alcohol use and other substance use in the Australian population. Drug Alcohol Rev. 2003;22(1):7–13.

Power C, Rodgers B, Hope S. U-shaped relation for alcohol consumption and health in early adulthood and implications for mortality. Lancet. 1998;352(9131):877.

Organization WH. The World health report: 2002: Reducing the risks, promoting healthy life. 2002.

Bien TH, Burge R. Smoking and drinking: a review of the literature. Subst Use Misuse. 1990;25(12):1429–54.

De Nooijer J, De Vries NK. Monitoring health risk behavior of Dutch adolescents and the development of health promoting policies and activities: the E-MOVO project. Health Promot Int. 2007;22(1):5–10.

De Nooijer J, Veling ML, Ton A, et al. Electronic monitoring and health promotion: an evaluation of the E-MOVO Web site by adolescents. Health Educ Res. 2008;23(3):382–91.

Statline C. Centraal Bureau voor de Statistiek. Gezondheidsmonitor; regio, bevolking van 19 jaar of ouder, 2012. Available from: http://statline.cbs.nl/Statweb/publication/?VW=T&DM=SLNL&PA=82166NED&D1=30-31&D2=0&D3=0&D4=a&D5=l&HD=161219-0108&HDR=T&STB=G1,G2,G3,G4 . Accessed 19 Dec 2016.

Statline C. Centraal Bureau voor de Statistiek. Gezondheidsmonitor; regio, bevolking van 19 jaar of ouder, 2012. Available from: http://statline.cbs.nl/Statweb/publication/?VW=T&DM=SLNL&PA=82166NED&D1=33-34&D2=0&D3=0&D4=a&D5=l&HD=161219-0126&HDR=T&STB=G1,G2,G3,G4 . (Accessed 19 Dec 2016).

Statline C. Centraal Bureau voor de Statistiek. Gezondheidsmonitor; regio, bevolking van 19 jaar of ouder, 2012. Available from: http://statline.cbs.nl/Statweb/publication/?VW=T&DM=SLNL&PA=71775NED&D1=25-27,31-32&D2=a&D3=18-19&D4=a&HD=161219-0212&HDR=T,G1&STB=G2,G3 . Accessed 19 Dec 2016.

O'malley PM, Bachman JG, Johnston LD. Reliability and consistency in self-reports of drug use. Subst Use Misuse. 1983;18(6):805–24.

Needle R, McCubbin H, Lorence J, et al. Reliability and validity of adolescent self-reported drug use in a family-based study: a methodological report. Subst Use Misuse. 1983;18(7):901–12.

Twente R. E-MOVO 2011: gezondheid, welzijn en leefstijl van jongeren in Twente. Twente: CHS; 2011.

Nijmegen C. E-MOVO 2011/2012 Gezondheid, welzijn en leefwijze van jongeren in de regio Nijmegen. Nijmegen: CHS; 2012.

Goodman R. The strengths and difficulties questionnaire: a research note. J Child Psychol Psychiatry. 1997;38(5):581–6.

Muris P, Meesters C, van den Berg F. The strengths and difficulties questionnaire (SDQ). Eur Child Adolesc Psychiatry. 2003;12(1):1–8.

Smedje H, Broman J-E, Hetta J, et al. Psychometric properties of a Swedish version of the “Strengths and Difficulties Questionnaire”. Eur Child Adolesc Psychiatry. 1999;8(2):63–70.

Truman J, Robinson K, Evans A, et al. The strengths and difficulties questionnaire. Eur Child Adolesc Psychiatry. 2003;12(1):9–14.

DeSalvo KB, Bloser N, Reynolds K, et al. Mortality prediction with a single general self-rated health question. J Gen Intern Med. 2006;21(3):267–75.

Article PubMed PubMed Central Google Scholar

Eaton DK, Kann L, Kinchen S, et al. Youth risk behavior surveillance--United States, 2007. Morb Mortal Wkly Rep Surveill Summ (Washington, DC: 2002). 2008;57(4):1–131.

Menard S. Applied logistic regression analysis. Thousand Oaks: Sage; 1995.

Myers RH. Classical and modern regression with applications (Duxbury Classic). Pacific Grove: Duxbury Press; 2000.

Baron RM, Kenny DA. The moderator–mediator variable distinction in social psychological research: conceptual, strategic, and statistical considerations. J Pers Soc Psychol. 1986;51(6):1173.

StatLine C. CBS Statline, Centraal Bureau voor de Statistiek. 2011.

Lahaut VM, Jansen HA, Van de Mheen D, et al. Non-response bias in a sample survey on alcohol consumption. Alcohol Alcohol. 2002;37(3):256–60.

Zhao J, Stockwell T, MacDonald S. Non–response bias in alcohol and drug population surveys. Drug Alcohol Rev. 2009;28(6):648–57.

McCartney K, Rosenthal R. Effect size, practical importance, and social policy for children. Child Dev. 2000;71(1):173–80.

Prentice DA, Miller DT. When small effects are impressive. Psychol Bull. 1992;112(1):160.

Markenstein L. Handreiking privacybescherming epidemiologie [Guide privacyprotection epidemiologie]. Utrecht: GGD Nederland; 2007.

College Bescherming Persoonsgegevens (CBP)): [the Dutch Data Protection Authority (Dutch DPA)], https://cbpweb.nl/ .

Download references

Acknowledgements

CS provided the Twente data. We are indebted to Annette Baltissen for providing the IJsselland data. The views expressed and any errors in this article are those of the authors and not of the CHS of Twente, the CHS of IJsselland, and the institutions the authors belong to.

No funding was acquired for this study.

Availability of data and materials

The datasets supporting the conclusions of this article are not publicly available in an online repository, but can be made available upon request. Requests should be directed at the CHS of Twente, Cees Smit ([email protected]).

Authors’ contributions

Regarding author contributions, KLC planned and managed the work, analysed and interpreted results and produced the first draft of the manuscript with support from MEP, PMK, and CS. Different versions of the manuscript have been reviewed and conceptualised by all co-authors. KLC produced the final manuscript and is the corresponding author. All authors have read and approved the final manuscript.

Competing interests

The authors declare that they have no competing interests.

Consent for publication

Not applicable.

Ethics approval and consent to participate

This study has been reported to the Dutch data protection authority and meets national ethics and privacy requirements. Parents were informed of the data collection by mail and they could refuse entry of their child into the data collection. This method of passive agreement is in accordance with Dutch legal standards [ 65 , 66 ].

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Author information

Authors and affiliations.

CAPHRI Care and Public Health Research Institute, Health Services Research, Maastricht University, Duboisdomein 30, 6229, GT, Maastricht, the Netherlands

Kei Long Cheung

Psychology, Health & Technology, University of Twente, Enschede, the Netherlands

Peter M. ten Klooster

CHS of Twente, Enschede, the Netherlands

CAPHRI Care and Public Health Research Institute, Health Promotion, Maastricht University, Maastricht, the Netherlands

Hein de Vries

Marcel E. Pieterse

You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Kei Long Cheung .

Rights and permissions

Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License ( http://creativecommons.org/licenses/by/4.0/ ), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated.

Reprints and permissions

About this article

Cite this article.

Cheung, K.L., ten Klooster, P.M., Smit, C. et al. The impact of non-response bias due to sampling in public health studies: A comparison of voluntary versus mandatory recruitment in a Dutch national survey on adolescent health. BMC Public Health 17 , 276 (2017). https://doi.org/10.1186/s12889-017-4189-8

Download citation

Received : 17 April 2016

Accepted : 16 March 2017

Published : 23 March 2017

DOI : https://doi.org/10.1186/s12889-017-4189-8

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

Non-response
sampling bias
health behaviour
recruitment
adolescents

BMC Public Health

ISSN: 1471-2458

Submission enquiries: [email protected]
General enquiries: [email protected]

Have a language expert improve your writing

Run a free plagiarism check in 10 minutes, generate accurate citations for free.

Knowledge Base
Research bias

Types of Bias in Research | Definition & Examples

Research bias results from any deviation from the truth, causing distorted results and wrong conclusions. Bias can occur at any phase of your research, including during data collection , data analysis , interpretation, or publication. Research bias can occur in both qualitative and quantitative research .

Understanding research bias is important for several reasons.

Bias exists in all research, across research designs , and is difficult to eliminate.
Bias can occur at any stage of the research process .
Bias impacts the validity and reliability of your findings, leading to misinterpretation of data.

It is almost impossible to conduct a study without some degree of research bias. It’s crucial for you to be aware of the potential types of bias, so you can minimize them.

For example, the success rate of the program will likely be affected if participants start to drop out ( attrition ). Participants who become disillusioned due to not losing weight may drop out, while those who succeed in losing weight are more likely to continue. This in turn may bias the findings towards more favorable results.

Information bias, interviewer bias.

Publication bias

Researcher bias

Response bias.

Selection bias

Cognitive bias

How to avoid bias in research

Other types of research bias, frequently asked questions about research bias.

Information bias , also called measurement bias, arises when key study variables are inaccurately measured or classified. Information bias occurs during the data collection step and is common in research studies that involve self-reporting and retrospective data collection. It can also result from poor interviewing techniques or differing levels of recall from participants.

The main types of information bias are:

Recall bias
Observer bias

Performance bias

Regression to the mean (rtm).

Over a period of four weeks, you ask students to keep a journal, noting how much time they spent on their smartphones along with any symptoms like muscle twitches, aches, or fatigue.

Recall bias is a type of information bias. It occurs when respondents are asked to recall events in the past and is common in studies that involve self-reporting.

As a rule of thumb, infrequent events (e.g., buying a house or a car) will be memorable for longer periods of time than routine events (e.g., daily use of public transportation). You can reduce recall bias by running a pilot survey and carefully testing recall periods. If possible, test both shorter and longer periods, checking for differences in recall.

A group of children who have been diagnosed, called the case group
A group of children who have not been diagnosed, called the control group

Since the parents are being asked to recall what their children generally ate over a period of several years, there is high potential for recall bias in the case group.

The best way to reduce recall bias is by ensuring your control group will have similar levels of recall bias to your case group. Parents of children who have childhood cancer, which is a serious health problem, are likely to be quite concerned about what may have contributed to the cancer.

Thus, if asked by researchers, these parents are likely to think very hard about what their child ate or did not eat in their first years of life. Parents of children with other serious health problems (aside from cancer) are also likely to be quite concerned about any diet-related question that researchers ask about.

Observer bias is the tendency of research participants to see what they expect or want to see, rather than what is actually occurring. Observer bias can affect the results in observationa l and experimental studies, where subjective judgment (such as assessing a medical image) or measurement (such as rounding blood pressure readings up or down) is part of the d ata collection process.

Observer bias leads to over- or underestimation of true values, which in turn compromise the validity of your findings. You can reduce observer bias by using double-blinded and single-blinded research methods.

Based on discussions you had with other researchers before starting your observations , you are inclined to think that medical staff tend to simply call each other when they need specific patient details or have questions about treatments.

At the end of the observation period, you compare notes with your colleague. Your conclusion was that medical staff tend to favor phone calls when seeking information, while your colleague noted down that medical staff mostly rely on face-to-face discussions. Seeing that your expectations may have influenced your observations, you and your colleague decide to conduct semi-structured interviews with medical staff to clarify the observed events. Note: Observer bias and actor–observer bias are not the same thing.

Performance bias is unequal care between study groups. Performance bias occurs mainly in medical research experiments, if participants have knowledge of the planned intervention, therapy, or drug trial before it begins.

Studies about nutrition, exercise outcomes, or surgical interventions are very susceptible to this type of bias. It can be minimized by using blinding , which prevents participants and/or researchers from knowing who is in the control or treatment groups. If blinding is not possible, then using objective outcomes (such as hospital admission data) is the best approach.

When the subjects of an experimental study change or improve their behavior because they are aware they are being studied, this is called the Hawthorne effect (or observer effect). Similarly, the John Henry effect occurs when members of a control group are aware they are being compared to the experimental group. This causes them to alter their behavior in an effort to compensate for their perceived disadvantage.

Regression to the mean (RTM) is a statistical phenomenon that refers to the fact that a variable that shows an extreme value on its first measurement will tend to be closer to the center of its distribution on a second measurement.

Medical research is particularly sensitive to RTM. Here, interventions aimed at a group or a characteristic that is very different from the average (e.g., people with high blood pressure) will appear to be successful because of the regression to the mean. This can lead researchers to misinterpret results, describing a specific intervention as causal when the change in the extreme groups would have happened anyway.

In general, among people with depression, certain physical and mental characteristics have been observed to deviate from the population mean .

This could lead you to think that the intervention was effective when those treated showed improvement on measured post-treatment indicators, such as reduced severity of depressive episodes.

However, given that such characteristics deviate more from the population mean in people with depression than in people without depression, this improvement could be attributed to RTM.

Interviewer bias stems from the person conducting the research study. It can result from the way they ask questions or react to responses, but also from any aspect of their identity, such as their sex, ethnicity, social class, or perceived attractiveness.

Interviewer bias distorts responses, especially when the characteristics relate in some way to the research topic. Interviewer bias can also affect the interviewer’s ability to establish rapport with the interviewees, causing them to feel less comfortable giving their honest opinions about sensitive or personal topics.

Participant: “I like to solve puzzles, or sometimes do some gardening.”

You: “I love gardening, too!”

In this case, seeing your enthusiastic reaction could lead the participant to talk more about gardening.

Establishing trust between you and your interviewees is crucial in order to ensure that they feel comfortable opening up and revealing their true thoughts and feelings. At the same time, being overly empathetic can influence the responses of your interviewees, as seen above.

Publication bias occurs when the decision to publish research findings is based on their nature or the direction of their results. Studies reporting results that are perceived as positive, statistically significant , or favoring the study hypotheses are more likely to be published due to publication bias.

Publication bias is related to data dredging (also called p -hacking ), where statistical tests on a set of data are run until something statistically significant happens. As academic journals tend to prefer publishing statistically significant results, this can pressure researchers to only submit statistically significant results. P -hacking can also involve excluding participants or stopping data collection once a p value of 0.05 is reached. However, this leads to false positive results and an overrepresentation of positive results in published academic literature.

Researcher bias occurs when the researcher’s beliefs or expectations influence the research design or data collection process. Researcher bias can be deliberate (such as claiming that an intervention worked even if it didn’t) or unconscious (such as letting personal feelings, stereotypes, or assumptions influence research questions ).

The unconscious form of researcher bias is associated with the Pygmalion effect (or Rosenthal effect ), where the researcher’s high expectations (e.g., that patients assigned to a treatment group will succeed) lead to better performance and better outcomes.

Researcher bias is also sometimes called experimenter bias, but it applies to all types of investigative projects, rather than only to experimental designs .

Good question: What are your views on alcohol consumption among your peers?
Bad question: Do you think it’s okay for young people to drink so much?

Response bias is a general term used to describe a number of different situations where respondents tend to provide inaccurate or false answers to self-report questions, such as those asked on surveys or in structured interviews .

This happens because when people are asked a question (e.g., during an interview ), they integrate multiple sources of information to generate their responses. Because of that, any aspect of a research study may potentially bias a respondent. Examples include the phrasing of questions in surveys, how participants perceive the researcher, or the desire of the participant to please the researcher and to provide socially desirable responses.

Response bias also occurs in experimental medical research. When outcomes are based on patients’ reports, a placebo effect can occur. Here, patients report an improvement despite having received a placebo, not an active medical treatment.

While interviewing a student, you ask them:

“Do you think it’s okay to cheat on an exam?”

Common types of response bias are:

Acquiescence bias

Demand characteristics.

Social desirability bias

Courtesy bias

Question-order bias

Extreme responding

Acquiescence bias is the tendency of respondents to agree with a statement when faced with binary response options like “agree/disagree,” “yes/no,” or “true/false.” Acquiescence is sometimes referred to as “yea-saying.”

This type of bias occurs either due to the participant’s personality (i.e., some people are more likely to agree with statements than disagree, regardless of their content) or because participants perceive the researcher as an expert and are more inclined to agree with the statements presented to them.

Q: Are you a social person?

People who are inclined to agree with statements presented to them are at risk of selecting the first option, even if it isn’t fully supported by their lived experiences.

In order to control for acquiescence, consider tweaking your phrasing to encourage respondents to make a choice truly based on their preferences. Here’s an example:

Q: What would you prefer?

A quiet night in
A night out with friends

Demand characteristics are cues that could reveal the research agenda to participants, risking a change in their behaviors or views. Ensuring that participants are not aware of the research objectives is the best way to avoid this type of bias.

On each occasion, patients reported their pain as being less than prior to the operation. While at face value this seems to suggest that the operation does indeed lead to less pain, there is a demand characteristic at play. During the interviews, the researcher would unconsciously frown whenever patients reported more post-op pain. This increased the risk of patients figuring out that the researcher was hoping that the operation would have an advantageous effect.

Social desirability bias is the tendency of participants to give responses that they believe will be viewed favorably by the researcher or other participants. It often affects studies that focus on sensitive topics, such as alcohol consumption or sexual behavior.

You are conducting face-to-face semi-structured interviews with a number of employees from different departments. When asked whether they would be interested in a smoking cessation program, there was widespread enthusiasm for the idea.

Note that while social desirability and demand characteristics may sound similar, there is a key difference between them. Social desirability is about conforming to social norms, while demand characteristics revolve around the purpose of the research.

Courtesy bias stems from a reluctance to give negative feedback, so as to be polite to the person asking the question. Small-group interviewing where participants relate in some way to each other (e.g., a student, a teacher, and a dean) is especially prone to this type of bias.

Question order bias

Question order bias occurs when the order in which interview questions are asked influences the way the respondent interprets and evaluates them. This occurs especially when previous questions provide context for subsequent questions.

When answering subsequent questions, respondents may orient their answers to previous questions (called a halo effect ), which can lead to systematic distortion of the responses.

Extreme responding is the tendency of a respondent to answer in the extreme, choosing the lowest or highest response available, even if that is not their true opinion. Extreme responding is common in surveys using Likert scales , and it distorts people’s true attitudes and opinions.

Disposition towards the survey can be a source of extreme responding, as well as cultural components. For example, people coming from collectivist cultures tend to exhibit extreme responses in terms of agreement, while respondents indifferent to the questions asked may exhibit extreme responses in terms of disagreement.

Selection bias is a general term describing situations where bias is introduced into the research from factors affecting the study population.

Common types of selection bias are:

Sampling or ascertainment bias

Attrition bias
Self-selection (or volunteer) bias
Survivorship bias
Nonresponse bias
Undercoverage bias

Sampling bias occurs when your sample (the individuals, groups, or data you obtain for your research) is selected in a way that is not representative of the population you are analyzing. Sampling bias threatens the external validity of your findings and influences the generalizability of your results.

The easiest way to prevent sampling bias is to use a probability sampling method . This way, each member of the population you are studying has an equal chance of being included in your sample.

Sampling bias is often referred to as ascertainment bias in the medical field.

Attrition bias occurs when participants who drop out of a study systematically differ from those who remain in the study. Attrition bias is especially problematic in randomized controlled trials for medical research because participants who do not like the experience or have unwanted side effects can drop out and affect your results.

You can minimize attrition bias by offering incentives for participants to complete the study (e.g., a gift card if they successfully attend every session). It’s also a good practice to recruit more participants than you need, or minimize the number of follow-up sessions or questions.

You provide a treatment group with weekly one-hour sessions over a two-month period, while a control group attends sessions on an unrelated topic. You complete five waves of data collection to compare outcomes: a pretest survey, three surveys during the program, and a posttest survey.

Self-selection or volunteer bias

Self-selection bias (also called volunteer bias ) occurs when individuals who volunteer for a study have particular characteristics that matter for the purposes of the study.

Volunteer bias leads to biased data, as the respondents who choose to participate will not represent your entire target population. You can avoid this type of bias by using random assignment —i.e., placing participants in a control group or a treatment group after they have volunteered to participate in the study.

Closely related to volunteer bias is nonresponse bias , which occurs when a research subject declines to participate in a particular study or drops out before the study’s completion.

Considering that the hospital is located in an affluent part of the city, volunteers are more likely to have a higher socioeconomic standing, higher education, and better nutrition than the general population.

Survivorship bias occurs when you do not evaluate your data set in its entirety: for example, by only analyzing the patients who survived a clinical trial.

This strongly increases the likelihood that you draw (incorrect) conclusions based upon those who have passed some sort of selection process—focusing on “survivors” and forgetting those who went through a similar process and did not survive.

Note that “survival” does not always mean that participants died! Rather, it signifies that participants did not successfully complete the intervention.

However, most college dropouts do not become billionaires. In fact, there are many more aspiring entrepreneurs who dropped out of college to start companies and failed than succeeded.

Nonresponse bias occurs when those who do not respond to a survey or research project are different from those who do in ways that are critical to the goals of the research. This is very common in survey research, when participants are unable or unwilling to participate due to factors like lack of the necessary skills, lack of time, or guilt or shame related to the topic.

You can mitigate nonresponse bias by offering the survey in different formats (e.g., an online survey, but also a paper version sent via post), ensuring confidentiality , and sending them reminders to complete the survey.

You notice that your surveys were conducted during business hours, when the working-age residents were less likely to be home.

Undercoverage bias occurs when you only sample from a subset of the population you are interested in. Online surveys can be particularly susceptible to undercoverage bias. Despite being more cost-effective than other methods, they can introduce undercoverage bias as a result of excluding people who do not use the internet.

Cognitive bias refers to a set of predictable (i.e., nonrandom) errors in thinking that arise from our limited ability to process information objectively. Rather, our judgment is influenced by our values, memories, and other personal traits. These create “ mental shortcuts” that help us process information intuitively and decide faster. However, cognitive bias can also cause us to misunderstand or misinterpret situations, information, or other people.

Because of cognitive bias, people often perceive events to be more predictable after they happen.

Although there is no general agreement on how many types of cognitive bias exist, some common types are:

Anchoring bias
Framing effect
Actor-observer bias
Availability heuristic (or availability bias)
Confirmation bias
Halo effect
The Baader-Meinhof phenomenon

Anchoring bias

Anchoring bias is people’s tendency to fixate on the first piece of information they receive, especially when it concerns numbers. This piece of information becomes a reference point or anchor. Because of that, people base all subsequent decisions on this anchor. For example, initial offers have a stronger influence on the outcome of negotiations than subsequent ones.

Framing effect

Framing effect refers to our tendency to decide based on how the information about the decision is presented to us. In other words, our response depends on whether the option is presented in a negative or positive light, e.g., gain or loss, reward or punishment, etc. This means that the same information can be more or less attractive depending on the wording or what features are highlighted.

Actor–observer bias

Actor–observer bias occurs when you attribute the behavior of others to internal factors, like skill or personality, but attribute your own behavior to external or situational factors.

In other words, when you are the actor in a situation, you are more likely to link events to external factors, such as your surroundings or environment. However, when you are observing the behavior of others, you are more likely to associate behavior with their personality, nature, or temperament.

One interviewee recalls a morning when it was raining heavily. They were rushing to drop off their kids at school in order to get to work on time. As they were driving down the highway, another car cut them off as they were trying to merge. They tell you how frustrated they felt and exclaim that the other driver must have been a very rude person.

At another point, the same interviewee recalls that they did something similar: accidentally cutting off another driver while trying to take the correct exit. However, this time, the interviewee claimed that they always drive very carefully, blaming their mistake on poor visibility due to the rain.

Availability heuristic

Availability heuristic (or availability bias) describes the tendency to evaluate a topic using the information we can quickly recall to our mind, i.e., that is available to us. However, this is not necessarily the best information, rather it’s the most vivid or recent. Even so, due to this mental shortcut, we tend to think that what we can recall must be right and ignore any other information.

Confirmation bias

Confirmation bias is the tendency to seek out information in a way that supports our existing beliefs while also rejecting any information that contradicts those beliefs. Confirmation bias is often unintentional but still results in skewed results and poor decision-making.

Let’s say you grew up with a parent in the military. Chances are that you have a lot of complex emotions around overseas deployments. This can lead you to over-emphasize findings that “prove” that your lived experience is the case for most families, neglecting other explanations and experiences.

The halo effect refers to situations whereby our general impression about a person, a brand, or a product is shaped by a single trait. It happens, for instance, when we automatically make positive assumptions about people based on something positive we notice, while in reality, we know little about them.

The Baader-Meinhof phenomenon

The Baader-Meinhof phenomenon (or frequency illusion) occurs when something that you recently learned seems to appear “everywhere” soon after it was first brought to your attention. However, this is not the case. What has increased is your awareness of something, such as a new word or an old song you never knew existed, not their frequency.

While very difficult to eliminate entirely, research bias can be mitigated through proper study design and implementation. Here are some tips to keep in mind as you get started.

Clearly explain in your methodology section how your research design will help you meet the research objectives and why this is the most appropriate research design.
In quantitative studies , make sure that you use probability sampling to select the participants. If you’re running an experiment, make sure you use random assignment to assign your control and treatment groups.
Account for participants who withdraw or are lost to follow-up during the study. If they are withdrawing for a particular reason, it could bias your results. This applies especially to longer-term or longitudinal studies .
Use triangulation to enhance the validity and credibility of your findings.
Phrase your survey or interview questions in a neutral, non-judgmental tone. Be very careful that your questions do not steer your participants in any particular direction.
Consider using a reflexive journal. Here, you can log the details of each interview , paying special attention to any influence you may have had on participants. You can include these in your final analysis.
Baader–Meinhof phenomenon
Sampling bias
Ascertainment bias
Self-selection bias
Hawthorne effect
Omitted variable bias
Pygmalion effect
Placebo effect

Research bias affects the validity and reliability of your research findings , leading to false conclusions and a misinterpretation of the truth. This can have serious implications in areas like medical research where, for example, a new form of treatment may be evaluated.

Observer bias occurs when the researcher’s assumptions, views, or preconceptions influence what they see and record in a study, while actor–observer bias refers to situations where respondents attribute internal factors (e.g., bad character) to justify other’s behavior and external factors (difficult circumstances) to justify the same behavior in themselves.

Response bias is a general term used to describe a number of different conditions or factors that cue respondents to provide inaccurate or false answers during surveys or interviews. These factors range from the interviewer’s perceived social position or appearance to the the phrasing of questions in surveys.

Nonresponse bias occurs when the people who complete a survey are different from those who did not, in ways that are relevant to the research topic. Nonresponse can happen because people are either not willing or not able to participate.

Is this article helpful?

Other students also liked.

Attrition Bias | Examples, Explanation, Prevention
Observer Bias | Definition, Examples, Prevention
What Is Social Desirability Bias? | Definition & Examples

Refusal to participate in research among hard-to-reach populations: The case of detained persons

Roles Conceptualization, Data curation, Formal analysis, Funding acquisition, Investigation, Methodology, Project administration, Supervision, Writing – original draft

* E-mail: [email protected]

Affiliations Division of Prison Health, Geneva University Hospitals & University of Geneva, Geneva, Switzerland, Institute of Primary Health Care (BIHAM), University of Bern, Bern, Switzerland

Roles Conceptualization, Methodology, Writing – review & editing

Affiliations Division of Prison Health, Geneva University Hospitals & University of Geneva, Geneva, Switzerland, Department of Psychiatry, Geneva University Hospitals, Geneva, Switzerland

Affiliation Division of Prison Health, Geneva University Hospitals & University of Geneva, Geneva, Switzerland

Roles Conceptualization, Investigation, Methodology, Project administration, Writing – review & editing

Affiliations Division of Prison Health, Geneva University Hospitals & University of Geneva, Geneva, Switzerland, Division of Tropical and Humanitarian Medicine, Geneva University Hospitals, Geneva, Switzerland

Stéphanie Baggio,
Leonel Gonçalves,
Patrick Heller,
Hans Wolff,
Laurent Gétaz

Published: March 3, 2023
https://doi.org/10.1371/journal.pone.0282083
Reader Comments

Providing insights on refusal to participate in research is critical to achieve a better understanding of the non-response bias. Little is known on people who refused to participate, especially in hard-to-reach populations such as detained persons. This study investigated the potential non-response bias among detained persons, comparing participants who accepted or refused to sign a one-time general informed consent. We used data collected in a cross-sectional study primary designed to evaluate a one-time general informed consent for research. A total of 190 participants were included in the study (response rate = 84.7%). The main outcome was the acceptance to sign the informed consent, used as a proxy to evaluate non-response. We collected sociodemographic variables, health literacy, and self-reported clinical information. A total of 83.2% of the participants signed the informed consent. In the multivariable model after lasso selection and according to the relative bias, the most important predictors were the level of education (OR = 2.13, bias = 20.7%), health insurance status (OR = 2.04, bias = 7.8%), need of another study language (OR = 0.21, bias = 39.4%), health literacy (OR = 2.20, bias = 10.0%), and region of origin (not included in the lasso regression model, bias = 9.2%). Clinical characteristics were not significantly associated with the main outcome and had low relative biases (≤ 2.7%). Refusers were more likely to have social vulnerabilities than consenters, but clinical vulnerabilities were similar in both groups. The non-response bias probably occurred in this prison population. Therefore, efforts should be made to reach this vulnerable population, improve participation in research, and ensure a fair and equitable distribution of research benefits.

Citation: Baggio S, Gonçalves L, Heller P, Wolff H, Gétaz L (2023) Refusal to participate in research among hard-to-reach populations: The case of detained persons. PLoS ONE 18(3): e0282083. https://doi.org/10.1371/journal.pone.0282083

Editor: Andrea Knittel, University of North Carolina at Chapel Hill, UNITED STATES

Received: June 21, 2022; Accepted: February 2, 2023; Published: March 3, 2023

Copyright: © 2023 Baggio et al. This is an open access article distributed under the terms of the Creative Commons Attribution License , which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.

Data Availability: Data cannot be shared publicly because they contain sensitive information: Information on detained persons with a limited sample size that may allow individuals’ identification. Data are available from the corresponding author for researchers who meet the criteria for access to confidential data. Alternatively, requests can be sent to the Division of Prison Health of the Geneva University Hospitals (see https://www.hug.ch/medecine-penitentiaire ).

Funding: This study was funded by the University of Geneva (Mimosa funding) (SB). he funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.

Competing interests: The authors have declared that no competing interests exist.

Introduction

The non-response bias is a bias that occurs due to systematic differences between respondents and non-respondents. To better understand the non-response bias, insights on response rates and refusal to participate in research are needed, e.g., to assess whether participants are representative of the target population and to find out whether estimates are reliable. To date, little is known on people who refuse to participate, precisely because they decline participation. This is especially true for some hard-to-reach and vulnerable populations, such as detained persons.

Previous studies used late respondents as a proxy for non-respondents or compared respondents to non-respondents when information is available from the census population. For example, in their well-designed study in the substance use field, Studer et al. [ 1 ] concluded that non-respondents, late respondents, and respondents were significantly different from each other on substance use variables, with a non-negligible non-response bias. Other studies focusing on health-related topics, such as psychiatry [ 2 , 3 ], suicide [ 4 ], chronic diseases [ 5 ], and sexually transmitted diseases [ 6 , 7 ] obtained similar conclusions.

Overall, non-respondents appeared as more vulnerable than respondents. In the aforementioned studies, they displayed higher rates of substance use, psychiatric morbidity [ 1 – 4 , 8 ], severe chronic and infectious diseases [ 5 – 7 ], and lower health literacy prevalence estimates [ 9 ]. In addition, non-respondents usually display specific socioeconomic characteristics. For instance, they are more likely to come from deprived backgrounds and low socioeconomic strata [ 7 , 10 – 12 ].

Unfortunately, we know little about the non-response bias in detained populations, a vulnerable hard-to-reach population, with a severe burden of diseases and barriers to health care [ 13 – 16 ]. This study therefore aimed to investigate the potential non-response bias among detained persons, comparing participants who accepted to sign a one-time general informed consent to those who refused. The one-time informed consent was used as a proxy to understand the non-response bias, as information on detained persons who decline study participation is usually not available.

Materials and methods

Design and setting.

We used data collected in a study primary designed to evaluate a one-time general informed consent for research [ 17 ]. The one-time general informed consent for research allows using routinely collected data from the hospital’s medical files [ 18 ].

The study had a cross-sectional design with a parallel randomization (allocation 1:1 for groups reading a paper version of the informed consent or watching a video, see link in S1 File ). The study took place between December 2019 and December 2020 in the largest pre-trial Swiss prison (398 places), located in Geneva, in the French-speaking part of Switzerland, among male detained persons.

Participants provided oral consent to participate in the study, because the outcome of the main study was to provide a written one-time informed consent for research. Participants could refuse study participation. They were informed that they were free to participate and could refuse to participate or discontinue at any time, without any health care- or prison-related consequences. Participants received an incentive of CHF 20.- (~ 20€) for study participation (including those who did not signed the one-time informed consent). They were informed there was an incentive when they were invited to participate. The Geneva’s cantonal ethics committee approved the study protocol (no. 2019–01797), including the separate oral consent for study participation. Oral consent was registered on a separate file before participants’ randomization.

Participants

A total of 228 adult men were invited to participate (oral consent), of which n = 193 accepted (response rate = 84.7%). Three participants dropped out before the end of the study (i.e., they did not sign the one-time informed consent and did not respond to the 15-minute questionnaire), which left a final sample of 190 participants. Among these 190 participants, 158 (83.2%) signed the one-time informed consent and 32 (16.8%) declined. Refusers nonetheless consented to study participation and completed the questionnaire. The only exclusion criterion was having severe acute psychiatric issues that did not allow informed consent.

A study member from the prison medical unit conducted data collection (enrollment of participants, consent process, and 15-minute questionnaire), supervised by SB and LG. The study was conducted independently from the prison authorities and the prison staff was not involved. Study participation was offered to eligible participants visiting the prison medical unit (about 75% of persons detained in the current prison), while they were in the waiting room.

Participants either read a booklet or saw a video (the experimental condition in the primary study) and were invited to sign the informed consent (see S1 File ). Then, they all completed a 15-minute face-to-face questionnaire (including participants who refused to sign the informed consent). Participants first answered questions about the informed consent (questions not included in this study [ 17 ]) and then questions about sociodemographic and clinical variables.

The informed consent and questionnaire were available in the ten most common languages spoken in the prison: Albanian, Arabic, English, French, Georgian, German, Italian, Portuguese, Romanian, Russian, and Spanish.

Acceptance to sign the informed consent.

Participants could either agree (consenters) or decline (refusers) to sign the informed consent (legal Swiss document, see: https://swissethics.ch/en/templates/studieninformationen-und-einwilligungen ). This variable was used as a proxy to assess the non-response bias.

Sociodemographic variables.

Participants provided information on age, region of origin (Switzerland or European Union [EU] versus other countries), level of education (primary versus secondary/tertiary level of education), legal status of residence in Switzerland (yes/no), and health insurance status (having an insurance or not, which is mandatory in Switzerland). We also registered the language participants selected for their participation in the study (French versus other) and whether they would have preferred to answer in another language (yes/no).

Health literacy.

Health literacy was assessed using the three-question Short Test of Functional Health Literacy (S-TOFHLA) [ 19 ]. As the scale was negatively skewed, we recoded data in two categories: High versus low/moderate health literacy.

Clinical information.

Participants self-reported the presence of psychiatric disorders and somatic illnesses.

Statistical analyses

PPT PowerPoint slide
PNG larger image
TIFF original image

https://doi.org/10.1371/journal.pone.0282083.t001

Descriptive statistics are reported in Table 1 . Participants were on average 35.0 ± 11.8 years. Most of them came from countries outside the EU (63.6%), had a secondary or tertiary level of education (87.2%), and had no legal status in Switzerland (56.2%). Half of them did not have a health insurance (50.0%). They reported a high health literacy (58.5%) and few health conditions (somatic: 39.7%, psychiatric: 20.1%). Most participants chose French (57.9%) and 5.3% reported the need for another language than the ten available for the present study. No detained persons declined participation because of language difficulties. Regarding the outcome variable, a total of 83.2% signed the informed consent, whereas 16.8% declined.

We found the highest relative biases for the need of another language (39.4% for the group who answered “yes”), level of education (20.7% for the group with a primary level of education), health literacy (10.0% for the group with a low level of health literacy), region of origin (|9.2|% for the group from Switzerland or UE), health insurance (|7.8|% for the group having a health insurance), and study language (6.8% for the group that chose other languages than French).

Associations between acceptance to sign the consent and covariates are presented in Table 2 . In bivariate models, consenters were more likely to come from Switzerland and EU (odd ratio [OR] = 2.75, p = .036), to have a secondary or tertiary level of education (OR = 3.23, p = .017,), to have a health insurance (OR = 2.72, p = .020), and to have a high health literacy (OR = 2.90, p = .010). They were less likely to need another study language (OR = 0.18, p = .009). There were no differences between consenters and refusers for age (p = .778), legal status in Switzerland (p = .211), and health problems (somatic: p = .496, psychiatric: p = .909). In the multivariate model after lasso selection, statistically significant predictors were level of education (OR = 2.13), health insurance status (OR = 2.04), need of another study language (OR = 0.21), and health literacy (OR = 2.20).

https://doi.org/10.1371/journal.pone.0282083.t002

This study investigated the potential non-response bias among detained persons. For this purpose, we tested whether detained persons who refused to sign a one-time general informed consent were different from those who accepted.

Study findings showed that participants who declined participation displayed specific features compared to those who accepted. They were more likely to have social vulnerabilities, including educational barriers (low level of education), language and cultural barriers (especially not speaking one of the ten most common languages in the prison, but also being a migrant from outside the EU, and not speaking the region’s language), and health care-related barriers (low level of health literacy and no health insurance). These findings are in line with previous studies reporting that non-respondents were more likely to come from low socioeconomic areas, including low educational levels and migration backgrounds [ 7 , 10 – 12 ] and that non-response bias is related to health literacy [ 9 ].

Detained persons are a population with severe social vulnerabilities, likely to come from deprived backgrounds [ 13 ]. Taken together, our findings suggested that detained persons who refused to participate in research were a more vulnerable subgroup of this already disadvantaged population.

To date, few studies investigated the association between non-response and health insurance coverage. In the USA, a study showed that non-respondents were less likely to have a health insurance coverage compared to respondents [ 22 ]. Other study findings suggested that health care use was lower in non-respondents, among people having a health insurance coverage [ 23 ]. Switzerland has a compulsory universal health care coverage, paid by the individuals. State subsidies are available to ensure that everyone can afford basic health insurance. However, in our study, half of the participants did not have a health insurance.

Another important study finding was that refusers did not display clinical vulnerability, with similar proportions of somatic and psychiatric health problems. This result was inconsistent with previous research findings in the general population, which concluded that non-respondents had more psychiatric morbidity and somatic illnesses [ 1 , 2 , 4 – 7 ]. Potential explanations of this study findings are related to the type of population and access to health care. First, the prison population might be younger than population-based studies. Therefore, detained persons are less likely to have some severe somatic chronic diseases that occur in middle age. Second, detained persons often lack access to (primary) health care before and during detention and may therefore have underdiagnosed and undertreated somatic and psychiatric diseases [ 24 ]. They are thus unaware of potential somatic or psychiatric issues. Indeed, in our sample, only 20.1% of participants self-reported mental health problems, which is probably an underestimation of the true prevalence rate of mental health problems in the prison. In a previous meta-analysis, 39.8% to 49.2% of detained persons were identified as suffering from mental illnesses [ 14 – 16 ].

Refusal to participate in health research is a well-known threat for study validity, but not only. It is also likely to reinforce existing health inequalities, as refusers display different characteristics and may have different study outcomes. Since refusal to participate is more frequent in vulnerable people and hard-to-reach populations, with high health needs [ 25 ] and lack of access to primary health care [ 26 ], they may be excluded from research benefits [ 27 ]. For example, the exclusion of vulnerable subgroups from clinical trials (e.g., elderly) limit the generalizability of findings, with insufficient data about positive or negative effects of treatments [ 28 ]. Such exclusion may hinder access to new treatment and high-quality care [ 27 , 28 ]. Including vulnerable populations in research is critical for evidence-based medicine: It would enhance external validity, provide a fair access to research benefits, and improve understanding of vulnerabilities [ 25 , 29 ].

Previous studies concluded that the non-response bias is less critical when basic sociodemographic variables of responders and non-responders are similar or when the proportion of responders exceeds 70% [ 30 ]. Our study findings suggested that with similar age and a response rate of more than 85%, non-response bias might still exist. The non-response bias should be reduced, with awareness that simple rules may not apply and efforts to encourage study participation. To improve study participation, strategies such as interview incentives, community contacts, use of different recruitment channels, and promote trusting relationships could be used [ 12 , 25 ]. Guidelines are also available to improve informed consent, such as the “teach-to-goal” consent [ 29 ]. It helps to achieve a voluntary and truly informed consent.

This study had some limitations. First, the study took place at the prison medical unit. Detained persons who did not seek for health care were therefore not included (~25% of detained persons). This subgroup of detained persons was probably heterogenous, including people who did not need health care, but also people with specific vulnerabilities (e.g., language barriers, anosognosia). Second, some eligible participants declined study participation and were therefore not included in the “refusal” group (15.3%). These non-respondents could have different characteristics and our results should therefore be interpreted cautiously. The one-time informed consent was used as a proxy of non-response bias, but further studies should include detained persons who completely refuse to participate. Third, the study had a small sample size in some subgroups (i.e., participants without a legal status in Switzerland), with a potential lack of power. Fourth, data were collected among detained men. To avoid an increased gender bias in research [ 31 ], future studies should include women, even if detained persons are mostly men. Fifth, we relied on self-reports, which limited the reliability of clinical information. Use of census, administrative, and medical encounter data would be helpful to describe consenters and refusers more accurately. In addition, the reliability of self-reported information for refusers was questionable. Sixth, the S-TOFHLA assesses ability of individuals to read and understand health-related information, but it does not assess numeracy, which is an important dimension for health literacy [ 32 ]. Further studies should include a more complete range of measures of health literacy. Seventh, given important differences in sociopolitical contexts, penal systems, and prison populations, these results cannot be generalized outside Switzerland.

To conclude, the non-response bias probably occurs in prison populations, as it is the case in the general population. Detained persons who declined research participation were more likely to have social vulnerabilities. Therefore, efforts should be made to reach this vulnerable population, minimize non-response, and ensure a fair and equitable distribution of research benefits [ 12 ].

Supporting information

https://doi.org/10.1371/journal.pone.0282083.s001

View Article
PubMed/NCBI
Google Scholar
22. Kauff J, Olsen R, Fraker T. Nonrespondents and nonresponse bias: Evidence from a survey of Former welfare recipients in Iowa. Washington DC, USA: Mathematica Policy Research, Inc., 2002 2002. Report No.: MPR Reference No.: 8217–909 and 8703–106.
30. Prince M. 9—Epidemiology. In: Wright P, Stern J, Phelan M, editors. Core Psychiatry (Third Edition). Oxford: W.B. Saunders; 2012. p. 115–29.
31. Upchurch M. Gender bias in research. Companion to Women’s and Gender Studies: John Wiley & Sons, Ltd; 2020. p. 139–54.

- Google Chrome

Intended for healthcare professionals

Access provided by Google Indexer
My email alerts
BMA member login
Username * Password * Forgot your log in details? Need to activate BMA Member Log In Log in via OpenAthens Log in via your institution

Search form

Advanced search
Search responses
Search blogs
Non-response bias...

Non-response bias versus response bias

Related content
Peer review
Philip Sedgwick , reader in medical statistics and medical education
1 Centre for Medical and Healthcare Education, St George’s, University of London, London, UK
p.sedgwick{at}sgul.ac.uk

Researchers used a postal questionnaire survey to investigate the career progression of NHS doctors. The questionnaire included details about past and current employment, future career plans, and when career milestones were reached. Analysis was confined to respondents working in the UK NHS (including those with an honorary NHS contract). The participants were all those who graduated from UK medical schools in 1977, 1988, and 1993. The questionnaire was sent to 10 344 graduates, of whom 7012 replied, giving a response rate of 68%. 1

Men and women were compared, with the aim of establishing whether female doctors were disadvantaged in pursuing careers in the NHS. The researchers reported that women did not progress as far or as fast as men. However, it was suggested this was not because women encountered direct discrimination but that it was a reflection of not having always worked full time. Nonetheless, the possibility that indirect discrimination—for example, that the lack of opportunities for part time work may have influenced choice of specialty—could not be ruled out.

Which of the following statements, if any, are true?

a) Response bias is the opposite of non-response bias in definition

b) Response bias is a systematic difference between the answers provided by the survey respondents and their actual experiences

c) The presence of non-response bias would have affected the external validity of the survey

d) Non-response bias would have been minimised by an increased response rate to the survey

Statements b , c, and d are true, whereas a is false.

The word “bias” is often misunderstood when used in research methodology, probably because in this context it has a different meaning from its everyday usage, where it is used to imply “prejudice.” In the context of research, bias is the introduction of systematic error, subconsciously or otherwise, in the design, data collection, data analysis, or publication of a study.

Non-response bias and response bias are often confused. Response bias is not the opposite of non-response bias in definition ( a is false). Non-response bias would have occurred if there was a systematic difference in characteristics between responders and non-responders. Response bias would have occurred if there was a systematic difference in the way that respondents answered questions about their career progression, so that their answers did not accurately represent their experiences ( b is true).

The respondents to the survey were self selected and not a random sample from the three cohorts of graduates. This would have introduced non-response bias. The respondents would have been different to the non-responders in some way, not least in their motivation to complete the questionnaire. This would have ultimately affected the results of the survey. For example, doctors may have been less likely to return the questionnaire if they had become disillusioned with their career because its progression was not as fast as expected. This would have resulted in the survey underestimating the length of time doctors took to achieve their career milestones. The problem would have been exacerbated if the extent of non-response bias differed between men and women—the proposed subgroups for analysis in the above study. If non-response bias existed it would have threatened the external validity of the survey ( c is true)—that is, the extent to which the survey results could be generalised to the population of all UK NHS doctors.

Questionnaire surveys are prone to non-response bias. However, it may be difficult to quantify the extent of such bias because usually there is limited information, if any, about the characteristics, attitudes, and behaviour of those who do not respond. Obviously non-response bias can be minimised by ensuring that the response rate for a survey is as high as possible ( d is true).

Respondents to the survey gave details about their career milestones, including appointment as hospital consultant or general practice principal, and the date when this was first achieved. Response bias would have occurred if there was a systematic difference, subconsciously or otherwise, between the doctors’ responses and their actual experiences ( b is true). Response bias may have occurred for a variety of reasons. For example, respondents may have answered the questions about career progression in a way that they perceived was of interest to the researchers. Alternatively, some doctors may have wanted to promote some desired objective of their own and, for example, underestimated their career progression and gave dates later than when they actually achieved their career milestones. More generally, response bias is a particular problem in questionnaire surveys that investigate socially unacceptable or embarrassing behaviours, such as excessive alcohol consumption or drug taking.

The participants were all those who graduated from UK medical schools in 1977, 1988, and 1993. The researchers recognised that the career progression of doctors may have differed between the cohorts because of changes in working hours and reforms to specialist training. Analysis included investigation of the separate cohorts. However, it was acknowledged that it was unclear what effects, if any, the changes in working hours and training had on the results of the survey.

Response bias is one of a group of biases collectively known as ascertainment bias and sometimes referred to as detection bias. Ascertainment bias is the systematic distortion of the assessment of outcome measures by researchers or study participants. This group of biases is a particular problem in clinical trials when the researchers or participants are aware of the treatment allocation. 2

Cite this as: BMJ 2014;348:g2573

Competing interests: None declared.

↵ Taylor KS, Lambert TW, Goldacre MJ. Career progression and destinations, comparing men and women in the NHS: postal questionnaire surveys. BMJ 2009 ; 338 : b1735 . OpenUrl Abstract / FREE Full Text
↵ Sedgwick P. Bias in clinical trials. BMJ 2011 ; 343 : d4176 . OpenUrl FREE Full Text

UNC Chapel Hill

Join Our Listserv Request a Consult

Confronting Bias in Medical Algorithms at UNC Health

March 27, 2024

By Kriste Kuczynski

The fourth seminar in the Carolina Seminar Series: “ Science meets society at the intersection between precision medicine and justice, equity, and inclusion . ”

In person Event

Date: Wednesday, April 17

Time: 12:00 pm—1:15 pm

Location: MacNider 322

Registration required — https://go.unc.edu/sciencemeetssociety4

Lunch will be provided. Please place your order during registration before Friday April 12

This seminar features three UNC faculty who will offer their perspectives on bias in medical algorithms. Dr. Gibson, a nephrologist, will discuss race corrections for estimated glomerular filtration rate (eGFR). Dr. Goralski, Pulmonary/Critical Care, will discuss bias in pulse oximeter and pulmonology function tests. Dr. Seashore, Associate Chief Medical Informatics Officer for UNC Health, will discuss how UNC Health is approaching AI for clinical use, particularly when bias is “baked-in” to Epic@UNC.

Moderated by Gail Henderson, PhD

Featured Speakers:

Keisha L. Gibson, MD, MPH Associate Professor of Medicine and Pediatrics Chief, Pediatric Nephrology Division, UNC School of Medicine

To add this event to your calendar, you may download this iCal file .

Filed Under:

Developing non-response weights to account for attrition-related bias in a longitudinal pregnancy cohort

Tona M. Pitt 1 , 4 ,
Erin Hetherington 2 ,
Kamala Adhikari 3 , 4 ,
Shainur Premji 3 , 5 ,
Nicole Racine 6 , 7 ,
Suzanne C. Tough 1 , 3 &
Sheila McDonald 1 , 3 , 4

BMC Medical Research Methodology volume 23 , Article number: 295 ( 2023 ) Cite this article

532 Accesses

1 Altmetric

Metrics details

Prospective cohorts may be vulnerable to bias due to attrition. Inverse probability weights have been proposed as a method to help mitigate this bias. The current study used the “All Our Families” longitudinal pregnancy cohort of 3351 maternal-infant pairs and aimed to develop inverse probability weights using logistic regression models to predict study continuation versus drop-out from baseline to the three-year data collection wave.

Two methods of variable selection took place. One method was a knowledge-based a priori variable selection approach, while the second used Least Absolute Shrinkage and Selection Operator (LASSO). The ability of each model to predict continuing participation through discrimination and calibration for both approaches were evaluated by examining area under the receiver operating curve (AUROC) and calibration plots, respectively. Stabilized inverse probability weights were generated using predicted probabilities. Weight performance was assessed using standardized differences of baseline characteristics for those who continue in study and those that do not, with and without weights (unadjusted estimates).

The a priori and LASSO variable selection method prediction models had good and fair discrimination with AUROC of 0.69 (95% Confidence Interval [CI]: 0.67–0.71) and 0.73 (95% CI: 0.71–0.75), respectively. Calibration plots and non-significant Hosmer-Lemeshow Goodness of Fit Tests indicated that both the a priori (p = 0.329) and LASSO model (p = 0.242) were well-calibrated. Unweighted results indicated large (> 10%) standardized differences in 15 demographic variables (range: 11 − 29%), when comparing those who continued in the study with those that did not. Weights derived from the a priori and LASSO models reduced standardized differences relative to unadjusted estimates, with the largest differences of 13% and 5%, respectively. Additionally, when applying the same LASSO variable selection method to develop weights in future data collection waves, standardized differences remained below 10% for each demographic variable.

The LASSO variable selection approach produced robust weights that addressed non-response bias more than the knowledge-driven approach. These weights can be applied to analyses across multiple longitudinal waves of data collection to reduce bias.

Peer Review reports

Introduction

Longitudinal study designs allow researchers to establish temporality of exposure-outcome relationships by following samples of individuals over time with repeated measurements [ 1 ]. It is, however, common for participants in longitudinal cohorts to be lost to follow-up (i.e., attrition or censoring) [ 2 ]. While attrition over time is expected, it can contribute to biased exposure-outcome relationships depending on the nature of how and/or why individuals drop out of the study [ 3 ]. Attrition leaves researchers with challenges on how to address missing data, which will depend on why the data are missing, and has implications for analysis.

Several methods exist that aim to mitigate potential bias related to attrition. Complete case analysis and multiple imputation are used commonly, but both rely on assumptions related to how data are missing [ 4 ]. Attrition contributes to missing data that may be missing at random, missing not at random, or missing completely at random. Each changes the assumptions of how data are missing and the potential bias that may occur if one were to apply a complete case analysis [ 1 , 4 ]. Missing at random values are conditional on observed data, missing not at random depends on unobserved data, and missing completely at random depends on neither [ 1 , 4 ]. Another way to address attrition/censoring is to weight existing respondents using inverse probability of participation weights that are calculated based on the baseline information [ 5 , 6 , 7 ]. That is, the inverse of the probability of continuing in the study at subsequent waves of collection (i.e., those who have lower probability of continuing receive higher weights). This method accepts that individuals may drop out of longitudinal studies for various reasons and that these reasons can be modelled through weighting and using the existing data.

This study aimed to describe the process for developing and assessing the performance of weights in a pregnancy cohort that has spanned approximately 14 years to provide a statistical approach to account for attrition and the potential for selection bias. We describe two methods for developing a model to create the weights (one knowledge-based a priori model that is investigator derived and another data driven model using Least Absolute Shrinkage and Selection Operator [LASSO] regression), assessed the discrimination and calibration performance of each model, and then assessed the performance of the weights from each model. The best performing model was then applied to subsequent waves of data collection, and the performance of these weights was assessed to consider using these weights across all data collection waves in this cohort.

Cohort description

This study used the data from All Our Families Cohort (formerly All Our Babies Cohort) [ 8 ]. This is a pregnancy cohort that recruited 3387 women at less than 25 weeks gestational age in Calgary, Canada. Initial recruitment took place from May 2008 and December 2010 [ 8 ]. Women completed one survey at < 25 weeks gestational age, one at 34–36 weeks gestational age, and one at four months postpartum [ 8 ]. Four more surveys were conducted when their child reached one year (2009–2012), three years (2012–2014), five years (2014–2016), and eight years of age (2017–2019). Finally, a survey was conducted during the COVID-19 pandemic between May 20 and July 15, 2020 [ 9 , 10 ]. For the 8-year and COVID-19 surveys, data were collected and managed using Research Electronic Data Capture (REDCap) electronic data capture tools hosted at University of Calgary [ 11 , 12 ], prior to REDCap, data were collected using physical surveys and TeleForm to scan and verify data. This study used the first survey at < 25 weeks gestational age as the baseline cohort and the 3-year follow-up to assess non-participation. Three-year follow-up is chosen as there was little loss to follow-up in this cohort during gestation and at birth; at the 1-year follow-up, there were administrative challenges that affected response rate, but for reasons not related to general attrition. STATA 16.0 statistical software (StataCorp, College Station, TX, USA) was used for all analyses while the ggplot2 package in R software [ 13 ] was used to generate figures.

Model development

We examined two models: a priori and LASSO variable selection method, described below. For both models, the first survey was used to identify variables for inclusion in prediction models that ultimately led to weight development. The first survey included 127 variables across multiple topics, including socio-demographics, prenatal physical and mental health, lifestyle, and pre-pregnancy and life events. For both models, we used the follow-up survey conducted at three years for the outcome point (i.e., women who did not attend the 3-year survey were considered lost to follow-up). We later applied these models to subsequent waves of data collection (5-year follow-up, 8-year follow-up, and once during the COVID-19 pandemic).

The first method for weight generation followed a knowledge-based variable selection approach. Investigators with subject matter expertise in pregnancy cohorts (KA, SM, SP, TMP) met several times and collectively identified possible variables for inclusion, including possible interaction terms that could be related to drop-out over time. Decisions on which variables to include were based on existing content expertise as well as the quality of variable data (i.e., high proportion of missing data).

The second method followed a LASSO variable selection method described by Schmidt et al. [ 14 ]. This method was used to develop weights in a child cohort and uses least absolute shrinkage and selection operator (LASSO) regression to select relevant variables [ 15 ]. Categorical variables were left in categories as they were initially coded with the addition of a category for missing in some cases. For those categorical variables with missing data, missingness was recoded so that ‘missing’ became a category of the variable itself. If a single level within a categorical variable had a large coefficient based on the initial LASSO regression, we retained the overall categorical variable as a candidate for the next step of variable selection. This meant that continuous variables were cut into relevant categories and another level of “missing” was created. Next, we split the variables of interest into seven relevant context themes: Sociodemographic Characteristics, Pregnancy History, Conception History, Prenatal Care, Lifestyle and Health Care Use During Pregnancy, Mental Health/Social Support, and Smoking/Drug/Alcohol Consumption (current and previous). We applied LASSO regression with 10-fold cross validation to each context theme such that the tuning parameter minimizes the out-of-sample prediction error [ 14 , 16 ]. The three variables with the largest coefficients from the LASSO regression were fit in a multivariable logistic regression model and area under the receiver operating curve (ROC), sometimes referred to as C-statistic, was calculated from predicted probabilities. One at a time, the variable with the next largest coefficient was fit to the model and this process was repeated until the ROC was not significantly different (p > 0.05) from the previous ROC. This was completed for each context theme and all variables from each context them served as the candidate variables for the final model. Next, all of those top contributing variables, based on coefficient size, from each context theme were combined into a larger LASSO model. Only non-zero coefficients were selected for inclusion in the final logistic regression model.

Model assessment

We assessed the ability of the model to predict continuing participation through discrimination and calibration for both approaches. We assessed discrimination using area under the ROC. ROC plots are among the most common method of assessing discrimination and represents a curve of sensitivity over 1-specifictiy where sensitivity represents true positives (cases) while specificity represents true negatives (not cases) [ 17 ]. Values for ROC range from 0.5 (no better assessment than chance) to 1.0 (perfect discrimination). The following cut-offs are often suggested as guidelines to assess discrimination: ≤0.5 is no better than chance, > 0.5 and < 0.7 is poor, > 0.7 ≤ 0.8 is acceptable and > 0.8 is excellent [ 18 ]. Calibration of the model relates to the accuracy of predicted risk and has been defined as “for patients with a predicted risk of R%, on average R out of 100 should indeed suffer from the disease or event of interest” [ 19 ]. We assessed calibration through a combination of Hosmer-Lemeshow goodness-of-fit test, mean calibration, and calibration plots [ 20 ]. We then applied this model to the next wave of data collection (i.e., 5-year follow-up) to assess the temporal validity of the models and assess in the same way.

Weights assessment

Using the models derived from the a priori and LASSO variable selection method, we calculated predicted probabilities and stabilized inverse probability weights [ 21 ]. We applied stabilized weights as they typically result in less variance than non-stabilized weights [ 21 , 22 ]. The means, standard deviations (SDs), and ranges of the weights were calculated and plotted. Weights were truncated at the 0.5th and 99.5th percentiles to avoid bias due to extreme weights [ 23 ]. Weight performance was measured by comparing baseline characteristics of those who continued in the study and those who did not, with and without the weights. It has been suggested that the standardized difference is the preferred measure for comparing weight balance between groups (continued in study vs. lost to follow-up) and that a difference between groups of less than 10% is negligible [ 24 ]. We use the “pbalchk” package in STATA to calculate the standardized difference. Standardized differences can be calculated for continuous and categorical variables and involve both means for the continued and lost-to-follow up groups and their variances; for more information on this calculation see Austin, 2009 [ 25 ].

Finally, the same model identified at the 3-year follow-up was then re-fit to develop weights for each of the subsequent waves (i.e., 5-year follow-up, 8-year follow-up, and the first survey during the COVID-19 pandemic). The performance of these weights was assessed as above.

Based on the 3,351 singleton births in the All Our Families cohort, 1,990 (59.4%) continued participation at the three-year follow-up while 1,361 (40.6%) did not. Of note, the study population in follow-up waves differed slightly from baseline due to various reasons, such as child age eligibility for standardized developmental scales when data collection was initiated and associated funding and ethical constraints [ 9 ]. At the three-year follow-up 69% of participants from the two-year follow-up responded to the survey [ 9 ]. However, since some participants had dropped out at earlier waves (during pregnancy and at-birth waves), this represented 59% of the participants initially enrolled in the study. Ultimately, the a priori model contained 18 variables while the LASSO variable selection method model contained 22 (Table 1 ). The two models shared four variables (Education, Ethnicity, Physical Component Summary, and Previous History of Adverse Birth Outcomes).

The a priori model had poor-acceptable discrimination ROC of 0.69 (95% CI: 0.67–0.71) while the LASSO variable selection method model had acceptable discrimination ROC of 0.73 (95% CI: 0.71–0.75). Hosmer-Lemeshow goodness-of-fit tests with 10 bins were non-significant for both the knowledge-based (p = 0.329) and the LASSO variable selection method approach (p = 0.242). A statistically non-significant goodness-of-fit test indicates no statistical difference in the observed cases from the predicted cases [ 29 ]. A non-significant goodness-of-fit result implies a well-calibrated model; however, a goodness-of-fit test alone may not be sufficient to assess calibration [ 19 ]. To this end, we considered the mean calibration where “the average predicted risk is compared to the overall event rate” [ 19 ]. In this case, the ‘event rate’ is considered as the proportion of individuals who continue in the study at the 3-year follow-up and is compared with the calculated proportion derived from the a priori and LASSO variable selection method models. Mean calibrations were 0.594 (95% CI: 0.58–0.60) and 0.594 (95% CI: 0.59–0.60) for the a priori and LASSO variable selection method models, respectively, compared with an observed proportion of continued participation of 0.594. Given that the mean calibrations in the two models were very similar to that of the observed proportion, both models appeared well-calibrated. In addition, we visually examined the calibration plots for each model (Figs. 1 and 2 ). The a priori and LASSO variable selection method models were re-fit on the next wave of data collection (5-year follow-up) and performed similarly to the previous wave with ROC of 0.69 (95% CI: 0.67–0.71) and 0.73 (0.72–0.75), non-significant goodness-of-fit tests of p = 0.567 and p = 0.307, and mean calibration of 0.596 (95% CI: 0.59–0.60) and 0.593 (95% CI: 0.59–0.60), respectively.

Results of Calibration Curve for LASSO variable selection method Model; AUC: Area Under the Receiver Operating Curve

Results of Calibration Curve for a priori Model; AUC: Area Under the Receiver Operating Curve

In calibration plots, an ideal plot (a diagonal line with slope of 1 and intercept of 0) is presented with a calibration curve derived from the model data and demonstrates how similar (or not) the estimated risk is to observed risk. The plot is assessed by examining the curve slope (target of 1.0) and by using a loess function to compare curve of predicted risk with the ideal plot [ 18 ]. Both Hosmer-Lemeshow tests and mean calibration suggested moderate calibration, as did the calibration plots; although, the LASSO variable selection method model seemed more well-calibrated at higher values than the a priori model.

The stabilized weights for the a priori model had a mean (SD) of 1.00 (0.58) and a range of 0.43–10.1. The LASSO variable selection method model had a mean (SD) of 1.00 (0.74) and a range of 0.42–23.1. After trimming, the LASSO variable selection method and a priori models had maximum weights of 4.8 and 4.9, respectively. This resulted in changes to 33 individual’s weights in both models. As well, mean (SD) for the a priori and LASSO variable selection method models were 0.99 (0.46) and 0.99 (0.51), respectively.

Weights performance

The absolute standardized differences were calculated across baseline demographic variables (chosen a priori ) in the unweighted group were as large as 28.9% for home ownership and 27.5% for income (binary outcome split at $60,000) and a mean of standardized differences of 17.5%. In the a priori model, the largest absolute standardized difference was 13.1% (smoking history) with two variables having a standardized difference of 10% or greater and a mean of standardized differences of 4.6% (Fig. 3 ). In the LASSO variable selection method derived weights, the largest absolute standardized difference was just 5.4% (anxiety symptoms) with no variables greater than 10% and a mean of standardized differences of 2.5%. Comparisons of baseline characteristics are based on complete data at baseline; of the 15 variables measured, eight were missing data in ≤ 1% while the other seven (Income, Anxiety, Symptoms, Depression Symptoms, Maternal Age, New Canadian, and Household Size) ranged from 1 to 4.4%.

Comparing the unweighted absolute standardized differences with the stabilized truncated weights of a priori and LASSO variable selection method models

Since the LASSO variable selection method weights appeared to perform better, weights were developed using this approach and applied to subsequent waves of data collection with performance evaluated in the same way (Fig. 4 ). Across each follow-up wave of data collection (3-year, 5-year, 8-year, and COVID-19 survey [approximately 12-years of follow-up]), absolute standardized differences remained below 10% for baseline demographic variables.

Comparing absolute standardized differences with the stabilized truncated weights derived from LASSO variable selection method model across data collection follow-up waves (3-, 5-, 8-, year follow-up and follow-up during COVID-19) in longitudinal cohort

This study aimed to develop non-response weights for a pregnancy cohort that has followed participants for more than 12 years. To accomplish this, we examined two approaches: one a priori and another LASSO variable selection method. The LASSO variable selection method approach produced robust weights that addressed non-response bias more than the a priori approach. The data driven approach, however still required content knowledge in how data were grouped, combined, or split. These weights can be applied to analyses across multiple waves of data collection to reduce bias. While the a priori model performed well, the weights themselves did not reduce differences in baseline characteristics to the same degree as the LASSO variable selection method model. While the models contained different specific variables, there was some overlap in that variables between the models captured similar concepts. For example, the a priori model used the combined variable of ‘history of drug/alcohol dependence’ while the LASSO variable selection method model included drug use per week and number of alcoholic drinks per day. While both the a priori and LASSO variable selection method models had access to the same calculated variables, their component parts, and interaction terms, the a priori model attempted to create simplicity and reduce the number of variables within an overarching theme, the nuance of more specific variables was ultimately found to be more informative. As well, the a priori variable selection ended after initial selection of variables. Typically, in developing prediction models, the investigators would examine performance and re-calibrate as necessary, but for the purposes of variable selection performance this was not done. Further, the weights derived from the LASSO variable selection method approach were robust across waves. That is, the balance achieved at the 3-year follow-up was generally maintained through the 5-year and 8-year follow-up as well as through the survey during the COVID-19 pandemic (12 years after baseline). This indicates that the same factors influence retention over time, and that one model can be used to develop weights, and then applied consistently over several waves of data collection.

Unweighted differences in baseline characteristics existed with respect to attrition status during follow-up, suggesting the potential for selection bias. However, while bias due to attrition is possible in cohort studies, and should be considered, bias is not guaranteed simply due to differing baseline characteristics of those who continue those who drop-out if those differences do not exist between groups as they relate to the exposure-outcome relationship of interest. Previous work by this group has used weighted and unweighted results showed a slight difference in magnitude of results but no difference in trends [ 10 ]. Further, recent work has demonstrated no difference in results comparing modelled results of complete case analysis and inverse probability weighting using missing at random, missing not at random, and missing completely at random data [ 30 ]. To better understand the extent of bias due to attrition, comparison of analyses with and without weights is suggested. The weights created for this sample balanced demographic characteristics of those who continued participation and those who did not and serve as another way to quantitatively examine the potential role of attrition in creating bias in our longitudinal study cohort.

There exist some methodologic challenges in creating effective weights while also ensuring no undue influence of extreme weights. There is no clearly defined point at which to truncate weights but it is important to consider both heterogeneity in order to achieve balance and the role of extreme weights. The use of a very small amount of truncation seemed to be effective for this particular sample. By truncating just at the 99.5th percentile, we see the range in the LASSO variable selection method weights drop from 23.1 to 4.8 which would indicate just a few “outliers” that could have spuriously influenced the weights.

Strengths of this study include examining two approaches to developing weights and the comparison of the two. As well, this study used a large sample of over 3,351 participants with 127 individual variables that were considered. The breadth of variables allowed us to consider a multitude of factors that could predict continuation in the cohort in later waves.

This study is not without limitations. The LASSO variable selection method approach used missing data as a level within categories. This allowed us to maintain a large sample size, but it also meant that variables that would normally be continuous were categorized to create this missing level. Categorization of continuous variables can result in loss of information given the collapsing of participant data into groups.

This study outlined two approaches to developing non-response weights to address bias that may be introduced due to attrition, with a LASSO variable selection method approach creating weights performing better that a priori approach in balancing baseline characteristics. The All Our Families cohort observes approximately 60% of participants returning to the study eight years after giving birth, in line with other major pregnancy cohorts [ 31 , 32 , 33 ]. The use of inverse probability weights considers the potential effect of non-response bias and the weights developed here can be applied to future studies using the AOF cohort data in secondary analyses and subsequent data collections; a further advantage of the use of these weights is that they can be easily applied to a variety of outcome models (i.e., linear regression, logistic regression, survival analysis). Importantly, the approach used in the present study in creating these weights could be applied in other cohorts, where the potential for selection bias exists due to attrition. Balancing the characteristics of participants at later cohort data collection waves to the sample recruited at baseline increases the confidence that temporal associations better reflect the experience of the target population.

Data Availability

The datasets analysed during the current study are not publicly available as they contain personal participant information but are available from the corresponding author, through the All Our Families Cohort Study, on reasonable request.

Dohoo IR, Martin SW, Stryhn H. Methods in epidemiologic research. 2012.

Kristman V, Manno M, Côté P. Loss to follow-up in cohort studies: how much is too much? Eur J Epidemiol. 2004;19:751–60.

Article PubMed Google Scholar

Dettori JR. Loss to follow-up. Evidence-based spine-care Journal. 2011;2(01):7–10.

Article PubMed PubMed Central Google Scholar

Little RJ, Rubin DB. Statistical analysis with missing data. John Wiley & Sons; 2019. Apr 23.

Seaman SR, White IR. Review of inverse probability weighting for dealing with missing data. Stat Methods Med Res. 2013;22(3):278–95.

Biele G, Gustavson K, Czajkowski NO, Nilsen RM, Reichborn-Kjennerud T, Magnus PM, Stoltenberg C, Aase H. Bias from self selection and loss to follow-up in prospective cohort studies. Eur J Epidemiol. 2019;34(10):927–38.

Article CAS PubMed Google Scholar

Howe CJ, Cole SR, Lau B, Napravnik S, Eron JJ Jr. Selection Bias due to loss to follow up in Cohort studies. Epidemiology. 2016;27(1):91–7.

McDonald SW, Lyon AW, Benzies KM, McNeil DA, Lye SJ, Dolan SM, Pennell CE, Bocking AD, Tough SC. The all our babies pregnancy cohort: design, methods, and participant characteristics. BMC Pregnancy Childbirth. 2013;13(1):1–2.

Google Scholar

Tough SC, McDonald SW, Collisson BA, Graham SA, Kehler H, Kingston D, Benzies K. Cohort profile: the all our babies pregnancy cohort (AOB). Int J Epidemiol. 2017;46(5):1389–90k.

Racine N, Hetherington E, McArthur BA, McDonald S, Edwards S, Tough S, Madigan S. Maternal depressive and anxiety symptoms before and during the COVID-19 pandemic in Canada: a longitudinal analysis. The Lancet Psychiatry. 2021;8(5):405–15.

Harris PA, Taylor R, Thielke R, Payne J, Gonzalez N, Conde JG. Research electronic data capture (REDCap)—a metadata-driven methodology and workflow process for providing translational research informatics support. J Biomed Inform. 2009;42(2):377–81.

Harris PA, Taylor R, Minor BL, Elliott V, Fernandez M, O’Neal L, McLeod L, Delacqua G, Delacqua F, Kirby J, Duda SN. The REDCap consortium: building an international community of software platform partners. J Biomed Inform. 2019;95:103208.

Wickham H. ggplot2: elegant graphics for data analysis. Volume ISBN 978–3–319–24277–4. Springer-Verlag New York; 2016. https://ggplot2.tidyverse.org .

Schmidt SC, Woll A. Longitudinal drop-out and weighting against its bias. BMC Med Res Methodol. 2017;17(1):1–1.

Article Google Scholar

Tibshirani R. Regression shrinkage and selection via the lasso. J Roy Stat Soc: Ser B (Methodol). 1996;58(1):267–88.

Lee TF, Chao PJ, Ting HM, Chang L, Huang YJ, Wu JM, Wang HY, Horng MF, Chang CM, Lan JH, Huang YY. Using multivariate regression model with least absolute shrinkage and selection operator (LASSO) to predict the incidence of xerostomia after intensity-modulated radiotherapy for Head and Neck cancer. PLoS ONE. 2014;9(2):e89700.

Cook NR. Use and misuse of the receiver operating characteristic curve in risk prediction. Circulation. 2007;115(7):928–35.

Hosmer DW Jr, Lemeshow S, Sturdivant RX. Applied logistic regression. John Wiley & Sons; 2013 Apr. p. 1.

Van Calster B, Nieboer D, Vergouwe Y, De Cock B, Pencina MJ, Steyerberg EW. A calibration hierarchy for risk models was defined: from utopia to empirical data. J Clin Epidemiol. 2016;74:167–76.

Van Calster B, McLernon DJ, Van Smeden M, Wynants L, Steyerberg EW. Calibration: the Achilles heel of predictive analytics. BMC Med. 2019;17(1):1–7.

Robins JM, Hernan MA, Brumback B. Marginal structural models and causal inference in epidemiology. Epidemiology. 2000;11:550–60.

Chesnaye NC, Stel VS, Tripepi G, Dekker FW, Fu EL, Zoccali C, Jager KJ. An introduction to inverse probability of treatment weighting in observational research. Clin Kidney J. 2022;15(1):14–20.

Cole SR, Hernan MA. Constructing inverse probability weights for marginal structural models. Am J Epidemiol. 2008;168:656–64.

Austin PC, Stuart EA. Moving towards best practice when using inverse probability of treatment weighting (IPTW) using the propensity score to estimate causal treatment effects in observational studies. Stat Med. 2015;34(28):3661–79.

Austin PC. Using the standardized difference to compare the prevalence of a binary variable between two groups in observational research. Commun statistics-simulation Comput. 2009;38(6):1228–34.

Ware JE Jr, Kosinski M, Keller SD. A 12-Item short-form Health Survey: construction of scales and preliminary tests of reliability and validity. Med Care. 1996 Mar;1:220–33.

Sherbourne CD, Stewart AL. The MOS social support survey. Soc Sci Med. 1991;32(6):705–14.

Cohen S, Kamarck T, Mermelstein R. A global measure of perceived stress. J Health Social Behav 1983 Dec 1:385–96.

Kramer AA, Zimmerman JE. Assessing the calibration of mortality benchmarks in critical care: the Hosmer-Lemeshow test revisited. Crit Care Med. 2007;35(9):2052–6.

Metten MA, Costet N, Multigner L, Viel JF, Chauvet G. Inverse probability weighting to handle attrition in cohort studies: some guidance and a call for caution. BMC Med Res Methodol. 2022;22(1):1–5.

Piler P, Kandrnal V, Kukla L, Andrýsková L, Švancara J, Jarkovský J, Dušek L, Pikhart H, Bobák M, Klánová J. Cohort profile: the European longitudinal study of pregnancy and childhood (ELSPAC) in the Czech Republic. Int J Epidemiol. 2017;46(5):1379–f.

PubMed Google Scholar

Fraser A, Macdonald-Wallis C, Tilling K, Boyd A, Golding J, Davey Smith G, Henderson J, Macleod J, Molloy L, Ness A, Ring S. Cohort profile: the Avon Longitudinal Study of parents and children: ALSPAC mothers cohort. Int J Epidemiol. 2013;42(1):97–110.

Kooijman MN, Kruithof CJ, van Duijn CM, Duijts L, Franco OH, van IJzendoorn MH, de Jongste JC, Klaver CC, van der Lugt A, Mackenbach JP, Moll HA. The Generation R Study: design and cohort update 2017. Eur J Epidemiol. 2016;31:1243–64.

Download references

Acknowledgements

The authors acknowledge the contribution and support of All Our Families participants and All Our Families team members.

All Our Families was funded through Alberta Innovates Interdisciplinary Team Grant #200700595 and the Alberta Children’s Hospital Foundation. TMP is supported by a Canadian Institutes of Health Research Doctoral Award (#187531). The funding bodies played no role in the design of the study and collection, analysis, and interpretation of data and in writing the manuscript.

Author information

Authors and affiliations.

Department of Paediatrics, University of Calgary, 28 Oki Drive NW, Calgary, T3B 6A8, Canada

Tona M. Pitt, Suzanne C. Tough & Sheila McDonald

Department of Epidemiology, Biostatistics and Occupational Health, McGill University, 2001 McGill College, Montreal, H3A 1G1, Canada

Erin Hetherington

Department of Community Health Sciences, University of Calgary, 3280 Hospital Drive NW, Calgary, T2N 4Z6, Canada

Kamala Adhikari, Shainur Premji, Suzanne C. Tough & Sheila McDonald

Provincial Population and Public Health, Alberta Health Services, 10301 Southport Rd SW, Calgary, T2W 1S7, Canada

Tona M. Pitt, Kamala Adhikari & Sheila McDonald

Centre for Health Economics, University of York, Heslington, YO10 5DD, York, UK

Shainur Premji

School of Psychology, Faculty of Social Sciences, University of Ottawa, 136 Jean- Jacques Lussier, Ottawa, K1N 6N5, Canada

Nicole Racine

Children’s Hospital of Eastern Ontario Research Institute, 401 Smyth Rd, Ottawa, K1H 5B2, Canada

You can also search for this author in PubMed Google Scholar

Contributions

TMP drafted the main text of the manuscript and prepared all figures and tables. EH and NR developed the initial analysis plan and contributed to the background for this study. EH and TMP wrote the code for statistical analysis and TMP conducted the analysis. TMP, KA, SP, and SM contributed to the knowledge-based analysis and developed the final analysis plan. SCT and SM co-led the data procurement and ongoing data collection for the prospective cohort data used. All authors reviewed the manuscript. All authors reviewed the manuscript and contributed to the interpretation of findings.

Corresponding author

Correspondence to Sheila McDonald .

Ethics declarations

Ethics approval.

Ethics approval for this study provided by the Conjoint Health Research Ethics Board at the University of Calgary (REB13-0868) and was completed in accordance within the guidelines and regulations of the Conjoint Health Research Ethics Board. This includes obtaining informed consent from all subjects included in the current study.

Consent for publication

Not applicable.

Competing interests

The authors declare no competing interests or conflicts of interest.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Pitt, T.M., Hetherington, E., Adhikari, K. et al. Developing non-response weights to account for attrition-related bias in a longitudinal pregnancy cohort. BMC Med Res Methodol 23 , 295 (2023). https://doi.org/10.1186/s12874-023-02121-1

Download citation

Received : 10 April 2023

Accepted : 07 December 2023

Published : 14 December 2023

DOI : https://doi.org/10.1186/s12874-023-02121-1

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

Cohort studies
Inverse probability weights
Non-response weights
All our families

BMC Medical Research Methodology

ISSN: 1471-2288

General enquiries: [email protected]

Read our research on: Abortion | Podcasts | Election 2024

Regions & Countries

How u.s. muslims are experiencing the israel-hamas war.

Muslim Americans gather in front of New York City Hall while demonstrating in support of Palestinians after performing Friday prayers on Oct. 20, 2023. (Fatih Aktas/Anadolu via Getty Images)

Related: How U.S. Jews are experiencing the Israel-Hamas war

U.S. Muslims are more sympathetic to the Palestinian people than many other Americans are, despite the fact that relatively few Muslims in the United States are Palestinian themselves, according to a Pew Research Center survey conducted in February . And only about a quarter of Muslims in the survey identify as Arab or of Arab ancestry.

Muslim Americans are also highly critical of President Joe Biden’s handling of the war between Israel and Hamas.

Meanwhile, a majority of Muslim Americans (70%) say discrimination against Muslims in our society has increased since the start of the war , and about half (53%) say news about the war makes them feel afraid.

Here’s a closer look at these and other findings from our new survey.

Pew Research Center conducted this survey to explore Americans’ views on the Israel-Hamas war, including what role the United States should play in the conflict. We surveyed a total of 12,693 U.S. adults from Feb. 13 to 25, 2024. Most of the respondents (10,642) are members of Pew Research Center’s American Trends Panel, an online survey panel recruited through national random sampling of residential addresses, which gives nearly all U.S. adults a chance of selection.

The remaining 2,051 respondents are members of three other survey panels – Ipsos’ KnowledgePanel, SSRS’s Opinion Panel, and NORC at the University of Chicago’s AmeriSpeak Panel – who were interviewed because they identify as Jewish or Muslim.

We “oversampled” (i.e., interviewed a disproportionately large number of) Jews and Muslims to provide more reliable estimates of their views on the topics covered in this survey. But these groups are not overrepresented in the national estimates reported here, because we adjusted for the oversampling in the weighting of the data. The survey is weighted to be representative of the U.S. adult population by gender, race, ethnicity, partisan affiliation, education, religious affiliation and other categories. In total, 1,941 Jewish and 414 Muslim respondents participated in this survey.

While the sample design was identical for Jews and Muslims, the resulting sample sizes are different. There are two main reasons for this. The Jewish population in the United States is roughly double the size of the Muslim population . Consequently, national survey panels have roughly twice as many or more Jewish panelists as Muslim ones. In addition, decades of research on survey nonresponse has shown that some groups in the U.S. are more likely to participate in surveys than others. Generally speaking, Jewish adults are more likely to participate in surveys than Muslim adults.

The survey also included questions about where people were born and whether people identify as Arab or of Arab origin. Because of insufficient sample size, we are unable to analyze Arab Americans or Americans of Israeli or Palestinian descent separately.

For more information on how we conducted this survey, refer to the ATP’s Methodology and the Methodology for this analysis. Here are the questions on views and knowledge of the Israel-Hamas war used in this analysis, and on perceptions of discrimination since the war began .

How U.S. Muslims view America’s role in the war

A bar chart showing that most U.S. Muslims say Biden is favoring the Israelis too much.

Only 6% of Muslim adults believe that the U.S. is striking the right balance between the Israelis and Palestinians, according to the February survey.

Most Muslims (60%) instead say Biden is favoring the Israelis too much, while just 3% say he is favoring the Palestinians too much. Another 30% are not sure.

A bar chart showing that Muslims in the U.S. have equally unfavorable views of Biden and Trump.

Muslim Americans have been strongly Democratic in the past and remain so – 66% of Muslim registered voters in the survey identify with or lean toward the Democratic Party. (The survey includes 298 Muslim registered voters for an effective sample size of 94 and a margin of error of plus or minus 10.1 points.) But Biden’s handling of the war has led some U.S. Muslims to cast protest votes against him in Democratic primaries this year.

A bar chart showing that most U.S. Muslims strongly favor U.S. humanitarian aid to Palestinians.

Muslims’ views of Biden are broadly negative, according to our survey: Only 36% view him positively. In fact, Muslims’ views of Biden are broadly similar to their views of former President Donald Trump (35% favorable), despite the fact that most Muslims felt Trump was unfriendly toward Muslims when he was president.

In the current war between Israel and Hamas, 69% of Muslim Americans favor the U.S. providing humanitarian aid to help Palestinian civilians. In contrast, most Muslims (65%) oppose America providing military aid to Israel to help in its war against Hamas.

How U.S. Muslims see the Palestinian, Israeli people and their leaders

A bar chart showing that most U.S. Muslims say their sympathies lie entirely or mostly with Palestinians.

While around a third of Muslim Americans (32%) have some sympathy for both the Israeli people and the Palestinian people, nearly two-thirds (64%) say their sympathies lie either entirely or mostly with the Palestinian people. Among the larger American public, by comparison, relatively few adults (16%) are entirely or mostly sympathetic toward the Palestinian people.

When it comes to the Israeli government, only 10% of U.S. Muslims have a favorable view. In fact, Muslims are more likely to have a favorable view of Hamas (37%), which has controlled Gaza, than of the Israeli government. Still, 58% of Muslims have an unfavorable view of Hamas.

A slight majority of Muslims (59%) have a favorable opinion of the Palestinian Authority, which some experts have suggested may take control of the Gaza Strip if Hamas is removed from power. The Palestinian Authority governs the West Bank and has not had control over the Gaza Strip since Hamas won elections in 2006 .

A dot plot showing that most U.S. Muslims see Hamas negatively – but still more positively than they see the Israeli government.

How U.S. Muslims perceive discrimination in the U.S. since the start of the war

Most Muslim Americans (70%) believe discrimination against Muslims in our society has increased since the start of the Israel-Hamas war. A much smaller share of the U.S. public overall (38%) says the same.

A bar chart showing that 70% of U.S. Muslims say discrimination against them has risen since the Israel-Hamas war began.

How U.S. Muslims are engaging with and following the war

A dot plot showing that about half of U.S. Muslims say news about the Israel-Hamas war makes them feel afraid.

Muslim Americans are more likely than Americans overall to feel afraid when hearing or reading news about the war. Around half of Muslims (53%) say this, compared with 37% of all U.S. adults. Muslim Americans are also more likely than U.S. adults overall to feel exhausted when consuming news about the war.

Around four-in-ten Muslim Americans say they are following the war extremely or very closely, while another 27% are somewhat following it. Still, roughly a third of U.S. Muslims (32%) are not following the war too closely or at all. Jewish Americans, by comparison, are following the war much more closely, according to our survey: 61% say they are following it extremely or very closely and 11% say they are following it not too or not at all closely.

About a third of U.S. Muslims could not correctly identify Benjamin Netanyahu as the current prime minister of Israel. And about three-in-ten Muslims could not correctly identify Hamas as the group behind the Oct. 7 attack against Israel or knew that most of the deaths in the Israel-Hamas war have been among Palestinians and not Israelis. Even so, roughly seven-in-ten correctly answered each question.

Sign up for our weekly newsletter

Fresh data delivered Saturday mornings

Muslims are a growing presence in U.S., but still face negative views from the public

Republicans account for a small but steady share of u.s. muslims, video: being muslim in the u.s., the share of americans who leave islam is offset by those who become muslim, new estimates show u.s. muslim population continues to grow, most popular.

About Pew Research Center Pew Research Center is a nonpartisan fact tank that informs the public about the issues, attitudes and trends shaping the world. It conducts public opinion polling, demographic research, media content analysis and other empirical social science research. Pew Research Center does not take policy positions. It is a subsidiary of The Pew Charitable Trusts .

IMAGES

What is Non-Response Bias and How to Overcome It?
9+ Tips for Identifying and Avoiding Response Bias in Surveys
Non-Response Bias: Meaning, Definition, and Examples
(PDF) A simplified eye-opener on managing missing data and in
Should We Care About Non-Response Bias? by Elizabeth Horn
BIAS in Medical or Clinical Research and how to avoid it

VIDEO

Lecture 1: Bias In Medical Research
From Recognizing Implicit Bias to Creating Action
2.1 Objective 6: Identify Sampling Bias, the Nonresponse Bias, and the Response Bias of a Study
Medical writing bias in myeloma clinical research
Medical Students Treated Patients Equally, Regardless of Race or Income
Prof. Dr. Suresh Sharma on Interdisciplinary Collaborations: Bias, Diversity, Tech Findings

COMMENTS

Study Bias
Non-response bias refers to significant differences between individuals who respond and those who do not respond to a survey or questionnaire. It is not to be confused as being the opposite of response bias. ... Chalmers I, Altman DG. How can medical journals help prevent poor medical research? Some opportunities presented by electronic ...
What Is Nonresponse Bias?| Definition & Example
Nonresponse bias can occur when individuals who refuse to take part in a study, or who drop out before the study is completed, are systematically different from those who participate fully. Nonresponse prevents the researcher from collecting data for all units in the sample. It is a common source of error, particularly in survey research.
A framework for exploring non-response patterns over time in health
Background Most health surveys have experienced a decline in response rates. A structured approach to evaluate whether a decreasing - and potentially more selective - response over time biased estimated trends in health behaviours is lacking. We developed a framework to explore the role of differential non-response over time. This framework was applied to a repeated cross-sectional survey in ...
Bias in research
The aim of this article is to outline types of 'bias' across research designs, and consider strategies to minimise bias. Evidence-based nursing, defined as the "process by which evidence, nursing theory, and clinical expertise are critically evaluated and considered, in conjunction with patient involvement, to provide the delivery of optimum nursing care,"1 is central to the continued ...
Reducing bias and improving transparency in medical research: a
Interest disclosures for published research are frequently incomplete. 16,17 Conflict of interest recording and policies in institutions that host research are also poor, and in several cases journal editors, as well as researchers, have potential conflicts of interest. 18,19 Voluntary declarations from the pharmaceutical industry have been criticised as inadequate, due to the ability of ...
Preventing bias from selective non-response in population-based survey
Background Health researchers often use survey studies to examine associations between risk factors at one time point and health outcomes later in life. Previous studies have shown that missing not at random (MNAR) may produce biased estimates in such studies. Medical researchers typically do not employ statistical methods for treating MNAR. Hence, there is a need to increase knowledge about ...
Non-response bias versus response bias
Response bias is one of a group of biases collectively known as ascertainment bias and sometimes referred to as detection bias. Ascertainment bias is the systematic distortion of the assessment of outcome measures by researchers or study participants.
Quantifying possible bias in clinical and epidemiological studies with
Bias in epidemiological studies can adversely affect the validity of study findings. Sensitivity analyses, known as quantitative bias analyses, are available to quantify potential residual bias arising from measurement error, confounding, and selection into the study. Effective application of these methods benefits from the input of multiple parties including clinicians, epidemiologists, and ...
Evidence of non-response bias in the Press-Ganey patient satisfaction
More research is needed to assess non-response bias—including follow-up studies of non-respondents—in order to more accurately measure of patient satisfaction. ... (Adjusted OR = 0.623 for Trauma vs. Adult Reconstruction). The response rate to the Press-Ganey Medical Practice Survey of outpatient satisfaction is low in an orthopaedic ...
The impact of non-response bias due to sampling in public health
In public health monitoring of young people it is critical to understand the effects of selective non-response, in particular when a controversial topic is involved like substance abuse or sexual behaviour. Research that is dependent upon voluntary subject participation is particularly vulnerable to sampling bias. As respondents whose participation is hardest to elicit on a voluntary basis are ...
Nonresponse Bias: Definition & Reducing
Nonresponse bias is a common problem in survey research because it is virtually impossible to get a 100% response rate. In fact, most response rates are less than 50%, and researchers typically consider 30% to be "good.". In other words, a survey with a reasonable response rate might still have 70% of the sample who don't respond.
Assessing and adjusting for non-response in the Millennium Cohort
In conducting population-based surveys, it is important to thoroughly examine and adjust for potential non-response bias to improve the representativeness of the sample prior to conducting analyses of the data and reporting findings. This paper examines factors contributing to second stage survey non-response during the baseline data collection for the Millennium Cohort Family Study, a large ...
Types of Bias in Research
Response bias also occurs in experimental medical research. When outcomes are based on patients' reports, a placebo effect can occur. Here, patients report an improvement despite having received a placebo, not an active medical treatment. Example: Response bias. You are researching factors associated with cheating among college students.
Refusal to participate in research among hard-to-reach ...
Introduction. The non-response bias is a bias that occurs due to systematic differences between respondents and non-respondents. To better understand the non-response bias, insights on response rates and refusal to participate in research are needed, e.g., to assess whether participants are representative of the target population and to find out whether estimates are reliable.
Non-response bias versus response bias
Non-response bias versus response bias. Researchers used a postal questionnaire survey to investigate the career progression of NHS doctors. The questionnaire included details about past and current employment, future career plans, and when career milestones were reached. Analysis was confined to respondents working in the UK NHS (including ...
Analyzing Non-Response Bias in a Seroprevalence Study of Vaccine
The following is a summary of "Investigating sources of non-response bias in a population-based seroprevalence study of vaccine-preventable diseases in the Netherlands," published in the February 2024 issue of Infectious Diseases by Postema et al. Researchers conducted a retrospective study to evaluate the representativeness of PIENTER 3 (P3) and identify non-response bias sources and trends
Confronting Bias in Medical Algorithms at UNC Health
A Center of Excellence for ELSI Research. CENTER FOR GENOMICS & SOCIETY. Search_for: Search. Search this site Search UNC School of Medicine. ... Confronting Bias in Medical Algorithms at UNC Health. In person Event. Date: Wednesday, April 17. Time: 12:00 pm—1:15 pm. Location: MacNider 322.
Methodology
AAPOR Task Force on Address-based Sampling. 2016. "AAPOR Report: Address-based Sampling." ↩ Email [email protected]. ↩; Postcard notifications are sent to 1) panelists who have been provided with a tablet to take ATP surveys, 2) panelists who were recruited within the last two years, and 3) panelists recruited prior to the last two years who opt to continue receiving postcard ...
Developing non-response weights to account for attrition-related bias
Background Prospective cohorts may be vulnerable to bias due to attrition. Inverse probability weights have been proposed as a method to help mitigate this bias. The current study used the "All Our Families" longitudinal pregnancy cohort of 3351 maternal-infant pairs and aimed to develop inverse probability weights using logistic regression models to predict study continuation versus drop ...
Rising Numbers of Americans Say Jews, Muslims Face a Lot of
Pew Research Center surveys conducted on our American Trends Panel (ATP) always include Jews and Muslims. But these surveys do not always have enough Jewish or Muslim respondents to report their answers separately. This is because they make up relatively small shares of the U.S. adult population: Roughly 2% of Americans say their religion is Judaism, and 1% say their religion is Islam.
Israel-Hamas war
But Jewish adults under 35 are divided over Israel's military response: 52% say the way Israel has carried out the war has been acceptable, while 42% call it unacceptable, and 6% are unsure. Jews ages 50 and older are far more likely to say Israel's conduct of the war has been acceptable (68%).
US Muslims' experiences and views
We surveyed a total of 12,693 U.S. adults from Feb. 13 to 25, 2024. Most of the respondents (10,642) are members of Pew Research Center's American Trends Panel, an online survey panel recruited through national random sampling of residential addresses, which gives nearly all U.S. adults a chance of selection.

StatPearls [Internet].

Affiliations

In this Page

Related information

Similar articles in PubMed

Recent Activity

Log in using your username and password

You are here

Statistics from Altmetric.com

What is bias in relation to research and why is understanding bias important?

How is bias minimised when undertaken research?

Read the full text or download the PDF:

The impact of non-response bias due to sampling in public health studies: A comparison of voluntary versus mandatory recruitment in a Dutch national survey on adolescent health

Conclusions

Sampling methods

Participants

Demographics

Tobacco consumption

Alcohol consumption

Mental health

Subjective health status

School experiences

Sexual behaviour

Statistical analyses

Sample characteristics

Comparing demographic characteristics of both samples with population estimates

Effects of sampling bias on demographic characteristics

Effects of sampling bias on health related variables

Effects of sampling bias on within-subject analyses

Limitations

Abbreviations

Acknowledgements

Availability of data and materials

Authors’ contributions

Competing interests

Consent for publication

Ethics approval and consent to participate

Publisher’s Note

Author information

Corresponding author

Rights and permissions

About this article

Share this article

BMC Public Health

Have a language expert improve your writing

Types of Bias in Research | Definition & Examples

Table of contents

Researcher bias

How to avoid bias in research

Performance bias

Acquiescence bias

Courtesy bias

Extreme responding

Question order bias

Sampling or ascertainment bias

Self-selection or volunteer bias

Anchoring bias

Actor–observer bias

The Baader-Meinhof phenomenon

Is this article helpful?

More interesting articles

Refusal to participate in research among hard-to-reach populations: The case of detained persons

Introduction

Materials and methods

Participants

Acceptance to sign the informed consent.

Sociodemographic variables.

Health literacy.

Clinical information.

Statistical analyses

Supporting information

Search form

Non-response bias versus response bias

Confronting Bias in Medical Algorithms at UNC Health

The fourth seminar in the Carolina Seminar Series: “ Science meets society at the intersection between precision medicine and justice, equity, and inclusion . ”​

Featured Speakers:

More from CENTER FOR GENOMICS & SOCIETY

Developing non-response weights to account for attrition-related bias in a longitudinal pregnancy cohort

Introduction

Cohort description

The fourth seminar in the Carolina Seminar Series: “ Science meets society at the intersection between precision medicine and justice, equity, and inclusion . ”