phd thesis validity

Reliability and validity of your thesis

Describing the reliability and validity of your research is an important part of your thesis.

Students in higher professional education as well as for academic students are required to describe these, and both are usually discussed in your methodology.

We help students with this daily, because describing these research concepts is often not too hard, but using them? That’s a different story!

In this article we explain these concepts and give you tips on how to use them in your thesis. Good luck!

Reliability, an example

When you look up the term reliability in you research manual, you will often find different definitions.

In the end, what matters with reliability is that the results of your research match the actual situation as much as possible. When a fellow student would research your topic, you want him or her to obtain practically the same results. Only then can we speak of a reliable study and will your research be reproducible.

Reliability in quantitative research

Reliability in quantitative research is often expressed as a reliability percentage of 95% or 99%. Do you want to know how many respondents you need to achieve this reliability score? You can use a sample size calculator, for example using this link.

You can also test the reliability of questionnaires in SPSS by calculating the so-called Cronbach’s Alpha. If this value is .80 or higher, this indicates a high reliability.

Most literature indicates that a score of 0.70 or higher is also still reliable.

Watch out: you can’t just combine all questions (with different response categories)!

Contact us now…

Validity in quantitative research

The validity of quantitative research usually has to do with which questions you ask your respondents. Validity means that the results you find will be factually correct.

It is often best to draw up your questions after you have finished your literature review (which is also most reliable!). You can then better judge which concepts are important, and use these concepts to draw up different questions (indicators). While doing this, always keep in mind the goal of the study and your sub-questions.

We have previously discussed the methodological validity, but you can also – this applies especially to academic students – look at the statistical validity.  For this you check whether the used statistical models meet the underlying assumptions.

Reliability and validity in qualitative research

Reliability is just as important in qualitative research as it is in quantitative research. In discussing this, you need to describe certain things regarding the circumstances of your research, like for example the fact that a quiet location was chosen (so that respondents could not be disturbed), the attitude of the interviewer (respondents should not be influenced, so take on an open attitude, ask open question instead of guiding ones, stay silent in between, etc.). It is also important that you describe the  representativeness of your sample,  as the representativeness of the sample makes for better reliability. If you wish, for example, to map the wishes and needs of a company’s target group, try not just to include current customers in your study but also and especially potential new costumers.

Furthermore, it is good to discuss why you have specifically chosen this research method and what the pros and cons of this method are.

When using interviews, for example, you can choose between structured, semi-structured or in-depth interviews. The goal is to justify your choice as well as possible. (Tip: it often helps to describe the purpose of the interviews.)

As with quantitative research, validity in qualitative research is about drawing up the right questions/topics that really cover your subject matter. Here too you can make use of an operationalization model. The interview questions (with structured and semi-structured interviews) are often listed in an interview-guide, to help you go to an interview well prepared. With in-depth interviews it is common to use an item-list.

In this document you can also write down for yourself an introduction that you give each interviewee at the start of an interview, and you also mention the confidentiality. See the image below for an example of an interview-guide.

Finally you need to describe here how you are going to perform the analysis and whether you have for example recorded the interviews (and transcribed them).

Moreover, it is always good to present your interview-questions to an expert and to do some trial interviews if possible.

Describing reliability and validity in your thesis

To summarize, discuss as many aspects as possible that have led to the highest possible reliability of your study. Include at least the following:

Reliability of literature review (secondary research)

The definition of a literature review according to Topscriptie is:  An overview of already existing information on your subject (what is known already) that shows the context of and identifies ‘gaps’ in the literature, takes the form of a critical and coherent review and sheds light on the connections between previous studies.

Many students forget to describe how they have tried to keep the reliability of their literature review (desk research) as high as possible. It is important to discuss this in your thesis. Describe for example the following aspects:

With the above mentioned tips from Topscriptie you can get started yourself on describing the reliability and validity of your study in your thesis. Don’t forget to discuss the limits of your study in your  discussion . It actually makes your article more solid when you can show that you are aware of these. Don’t forget that every thesis and every study has its own problems with the reliability and validity.

Topscriptie has already helped more than 6,000 students!

Let us help you with your studies or graduation. Discover what we can do for you.

Call or WhatsApp a thesis supervisor

Winner of the best thesis agency in the Netherlands

• Privacy Policy

Buy Me a Coffee

Home » Validity – Types, Examples and Guide

Validity – Types, Examples and Guide

Table of Contents

Definition:

Validity refers to the extent to which a concept, measure, or study accurately represents the intended meaning or reality it is intended to capture. It is a fundamental concept in research and assessment that assesses the soundness and appropriateness of the conclusions, inferences, or interpretations made based on the data or evidence collected.

Research Validity

Research validity refers to the degree to which a study accurately measures or reflects what it claims to measure. In other words, research validity concerns whether the conclusions drawn from a study are based on accurate, reliable and relevant data.

Validity is a concept used in logic and research methodology to assess the strength of an argument or the quality of a research study. It refers to the extent to which a conclusion or result is supported by evidence and reasoning.

How to Ensure Validity in Research

Ensuring validity in research involves several steps and considerations throughout the research process. Here are some key strategies to help maintain research validity:

Clearly Define Research Objectives and Questions

Start by clearly defining your research objectives and formulating specific research questions. This helps focus your study and ensures that you are addressing relevant and meaningful research topics.

Use appropriate research design

Select a research design that aligns with your research objectives and questions. Different types of studies, such as experimental, observational, qualitative, or quantitative, have specific strengths and limitations. Choose the design that best suits your research goals.

Use reliable and valid measurement instruments

If you are measuring variables or constructs, ensure that the measurement instruments you use are reliable and valid. This involves using established and well-tested tools or developing your own instruments through rigorous validation processes.

Ensure a representative sample

When selecting participants or subjects for your study, aim for a sample that is representative of the population you want to generalize to. Consider factors such as age, gender, socioeconomic status, and other relevant demographics to ensure your findings can be generalized appropriately.

Address potential confounding factors

Identify potential confounding variables or biases that could impact your results. Implement strategies such as randomization, matching, or statistical control to minimize the influence of confounding factors and increase internal validity.

Minimize measurement and response biases

Be aware of measurement biases and response biases that can occur during data collection. Use standardized protocols, clear instructions, and trained data collectors to minimize these biases. Employ techniques like blinding or double-blinding in experimental studies to reduce bias.

Conduct appropriate statistical analyses

Ensure that the statistical analyses you employ are appropriate for your research design and data type. Select statistical tests that are relevant to your research questions and use robust analytical techniques to draw accurate conclusions from your data.

Consider external validity

While it may not always be possible to achieve high external validity, be mindful of the generalizability of your findings. Clearly describe your sample and study context to help readers understand the scope and limitations of your research.

Peer review and replication

Submit your research for peer review by experts in your field. Peer review helps identify potential flaws, biases, or methodological issues that can impact validity. Additionally, encourage replication studies by other researchers to validate your findings and enhance the overall reliability of the research.

Transparent reporting

Clearly and transparently report your research methods, procedures, data collection, and analysis techniques. Provide sufficient details for others to evaluate the validity of your study and replicate your work if needed.

Types of Validity

There are several types of validity that researchers consider when designing and evaluating studies. Here are some common types of validity:

Internal Validity

Internal validity relates to the degree to which a study accurately identifies causal relationships between variables. It addresses whether the observed effects can be attributed to the manipulated independent variable rather than confounding factors. Threats to internal validity include selection bias, history effects, maturation of participants, and instrumentation issues.

External Validity

External validity concerns the generalizability of research findings to the broader population or real-world settings. It assesses the extent to which the results can be applied to other individuals, contexts, or timeframes. Factors that can limit external validity include sample characteristics, research settings, and the specific conditions under which the study was conducted.

Construct Validity

Construct validity examines whether a study adequately measures the intended theoretical constructs or concepts. It focuses on the alignment between the operational definitions used in the study and the underlying theoretical constructs. Construct validity can be threatened by issues such as poor measurement tools, inadequate operational definitions, or a lack of clarity in the conceptual framework.

Content Validity

Content validity refers to the degree to which a measurement instrument or test adequately covers the entire range of the construct being measured. It assesses whether the items or questions included in the measurement tool represent the full scope of the construct. Content validity is often evaluated through expert judgment, reviewing the relevance and representativeness of the items.

Criterion Validity

Criterion validity determines the extent to which a measure or test is related to an external criterion or standard. It assesses whether the results obtained from a measurement instrument align with other established measures or outcomes. Criterion validity can be divided into two subtypes: concurrent validity, which examines the relationship between the measure and the criterion at the same time, and predictive validity, which investigates the measure’s ability to predict future outcomes.

Face Validity

Face validity refers to the degree to which a measurement or test appears, on the surface, to measure what it intends to measure. It is a subjective assessment based on whether the items seem relevant and appropriate to the construct being measured. Face validity is often used as an initial evaluation before conducting more rigorous validity assessments.

Importance of Validity

Validity is crucial in research for several reasons:

• Accurate Measurement: Validity ensures that the measurements or observations in a study accurately represent the intended constructs or variables. Without validity, researchers cannot be confident that their results truly reflect the phenomena they are studying. Validity allows researchers to draw accurate conclusions and make meaningful inferences based on their findings.
• Credibility and Trustworthiness: Validity enhances the credibility and trustworthiness of research. When a study demonstrates high validity, it indicates that the researchers have taken appropriate measures to ensure the accuracy and integrity of their work. This strengthens the confidence of other researchers, peers, and the wider scientific community in the study’s results and conclusions.
• Generalizability: Validity helps determine the extent to which research findings can be generalized beyond the specific sample and context of the study. By addressing external validity, researchers can assess whether their results can be applied to other populations, settings, or situations. This information is valuable for making informed decisions, implementing interventions, or developing policies based on research findings.
• Sound Decision-Making: Validity supports informed decision-making in various fields, such as medicine, psychology, education, and social sciences. When validity is established, policymakers, practitioners, and professionals can rely on research findings to guide their actions and interventions. Validity ensures that decisions are based on accurate and trustworthy information, which can lead to better outcomes and more effective practices.
• Avoiding Errors and Bias: Validity helps researchers identify and mitigate potential errors and biases in their studies. By addressing internal validity, researchers can minimize confounding factors and alternative explanations, ensuring that the observed effects are genuinely attributable to the manipulated variables. Validity assessments also highlight measurement errors or shortcomings, enabling researchers to improve their measurement tools and procedures.
• Progress of Scientific Knowledge: Validity is essential for the advancement of scientific knowledge. Valid research contributes to the accumulation of reliable and valid evidence, which forms the foundation for building theories, developing models, and refining existing knowledge. Validity allows researchers to build upon previous findings, replicate studies, and establish a cumulative body of knowledge in various disciplines. Without validity, the scientific community would struggle to make meaningful progress and establish a solid understanding of the phenomena under investigation.
• Ethical Considerations: Validity is closely linked to ethical considerations in research. Conducting valid research ensures that participants’ time, effort, and data are not wasted on flawed or invalid studies. It upholds the principle of respect for participants’ autonomy and promotes responsible research practices. Validity is also important when making claims or drawing conclusions that may have real-world implications, as misleading or invalid findings can have adverse effects on individuals, organizations, or society as a whole.

Examples of Validity

Here are some examples of validity in different contexts:

• Example 1: All men are mortal. John is a man. Therefore, John is mortal. This argument is logically valid because the conclusion follows logically from the premises.
• Example 2: If it is raining, then the ground is wet. The ground is wet. Therefore, it is raining. This argument is not logically valid because there could be other reasons for the ground being wet, such as watering the plants.
• Example 1: In a study examining the relationship between caffeine consumption and alertness, the researchers use established measures of both variables, ensuring that they are accurately capturing the concepts they intend to measure. This demonstrates construct validity.
• Example 2: A researcher develops a new questionnaire to measure anxiety levels. They administer the questionnaire to a group of participants and find that it correlates highly with other established anxiety measures. This indicates good construct validity for the new questionnaire.
• Example 1: A study on the effects of a particular teaching method is conducted in a controlled laboratory setting. The findings of the study may lack external validity because the conditions in the lab may not accurately reflect real-world classroom settings.
• Example 2: A research study on the effects of a new medication includes participants from diverse backgrounds and age groups, increasing the external validity of the findings to a broader population.
• Example 1: In an experiment, a researcher manipulates the independent variable (e.g., a new drug) and controls for other variables to ensure that any observed effects on the dependent variable (e.g., symptom reduction) are indeed due to the manipulation. This establishes internal validity.
• Example 2: A researcher conducts a study examining the relationship between exercise and mood by administering questionnaires to participants. However, the study lacks internal validity because it does not control for other potential factors that could influence mood, such as diet or stress levels.
• Example 1: A teacher develops a new test to assess students’ knowledge of a particular subject. The items on the test appear to be relevant to the topic at hand and align with what one would expect to find on such a test. This suggests face validity, as the test appears to measure what it intends to measure.
• Example 2: A company develops a new customer satisfaction survey. The questions included in the survey seem to address key aspects of the customer experience and capture the relevant information. This indicates face validity, as the survey seems appropriate for assessing customer satisfaction.
• Example 1: A team of experts reviews a comprehensive curriculum for a high school biology course. They evaluate the curriculum to ensure that it covers all the essential topics and concepts necessary for students to gain a thorough understanding of biology. This demonstrates content validity, as the curriculum is representative of the domain it intends to cover.
• Example 2: A researcher develops a questionnaire to assess career satisfaction. The questions in the questionnaire encompass various dimensions of job satisfaction, such as salary, work-life balance, and career growth. This indicates content validity, as the questionnaire adequately represents the different aspects of career satisfaction.
• Example 1: A company wants to evaluate the effectiveness of a new employee selection test. They administer the test to a group of job applicants and later assess the job performance of those who were hired. If there is a strong correlation between the test scores and subsequent job performance, it suggests criterion validity, indicating that the test is predictive of job success.
• Example 2: A researcher wants to determine if a new medical diagnostic tool accurately identifies a specific disease. They compare the results of the diagnostic tool with the gold standard diagnostic method and find a high level of agreement. This demonstrates criterion validity, indicating that the new tool is valid in accurately diagnosing the disease.

Where to Write About Validity in A Thesis

In a thesis, discussions related to validity are typically included in the methodology and results sections. Here are some specific places where you can address validity within your thesis:

Research Design and Methodology

In the methodology section, provide a clear and detailed description of the measures, instruments, or data collection methods used in your study. Discuss the steps taken to establish or assess the validity of these measures. Explain the rationale behind the selection of specific validity types relevant to your study, such as content validity, criterion validity, or construct validity. Discuss any modifications or adaptations made to existing measures and their potential impact on validity.

Measurement Procedures

In the methodology section, elaborate on the procedures implemented to ensure the validity of measurements. Describe how potential biases or confounding factors were addressed, controlled, or accounted for to enhance internal validity. Provide details on how you ensured that the measurement process accurately captures the intended constructs or variables of interest.

Data Collection

In the methodology section, discuss the steps taken to collect data and ensure data validity. Explain any measures implemented to minimize errors or biases during data collection, such as training of data collectors, standardized protocols, or quality control procedures. Address any potential limitations or threats to validity related to the data collection process.

Data Analysis and Results

In the results section, present the analysis and findings related to validity. Report any statistical tests, correlations, or other measures used to assess validity. Provide interpretations and explanations of the results obtained. Discuss the implications of the validity findings for the overall reliability and credibility of your study.

Limitations and Future Directions

In the discussion or conclusion section, reflect on the limitations of your study, including limitations related to validity. Acknowledge any potential threats or weaknesses to validity that you encountered during your research. Discuss how these limitations may have influenced the interpretation of your findings and suggest avenues for future research that could address these validity concerns.

Applications of Validity

Validity is applicable in various areas and contexts where research and measurement play a role. Here are some common applications of validity:

Psychological and Behavioral Research

Validity is crucial in psychology and behavioral research to ensure that measurement instruments accurately capture constructs such as personality traits, intelligence, attitudes, emotions, or psychological disorders. Validity assessments help researchers determine if their measures are truly measuring the intended psychological constructs and if the results can be generalized to broader populations or real-world settings.

Educational Assessment

Validity is essential in educational assessment to determine if tests, exams, or assessments accurately measure students’ knowledge, skills, or abilities. It ensures that the assessment aligns with the educational objectives and provides reliable information about student performance. Validity assessments help identify if the assessment is valid for all students, regardless of their demographic characteristics, language proficiency, or cultural background.

Program Evaluation

Validity plays a crucial role in program evaluation, where researchers assess the effectiveness and impact of interventions, policies, or programs. By establishing validity, evaluators can determine if the observed outcomes are genuinely attributable to the program being evaluated rather than extraneous factors. Validity assessments also help ensure that the evaluation findings are applicable to different populations, contexts, or timeframes.

Medical and Health Research

Validity is essential in medical and health research to ensure the accuracy and reliability of diagnostic tools, measurement instruments, and clinical assessments. Validity assessments help determine if a measurement accurately identifies the presence or absence of a medical condition, measures the effectiveness of a treatment, or predicts patient outcomes. Validity is crucial for establishing evidence-based medicine and informing medical decision-making.

Social Science Research

Validity is relevant in various social science disciplines, including sociology, anthropology, economics, and political science. Researchers use validity to ensure that their measures and methods accurately capture social phenomena, such as social attitudes, behaviors, social structures, or economic indicators. Validity assessments support the reliability and credibility of social science research findings.

Market Research and Surveys

Validity is important in market research and survey studies to ensure that the survey questions effectively measure consumer preferences, buying behaviors, or attitudes towards products or services. Validity assessments help researchers determine if the survey instrument is accurately capturing the desired information and if the results can be generalized to the target population.

Limitations of Validity

Here are some limitations of validity:

• Construct Validity: Limitations of construct validity include the potential for measurement error, inadequate operational definitions of constructs, or the failure to capture all aspects of a complex construct.
• Internal Validity: Limitations of internal validity may arise from confounding variables, selection bias, or the presence of extraneous factors that could influence the study outcomes, making it difficult to attribute causality accurately.
• External Validity: Limitations of external validity can occur when the study sample does not represent the broader population, when the research setting differs significantly from real-world conditions, or when the study lacks ecological validity, i.e., the findings do not reflect real-world complexities.
• Measurement Validity: Limitations of measurement validity can arise from measurement error, inadequately designed or flawed measurement scales, or limitations inherent in self-report measures, such as social desirability bias or recall bias.
• Statistical Conclusion Validity: Limitations in statistical conclusion validity can occur due to sampling errors, inadequate sample sizes, or improper statistical analysis techniques, leading to incorrect conclusions or generalizations.
• Temporal Validity: Limitations of temporal validity arise when the study results become outdated due to changes in the studied phenomena, interventions, or contextual factors.
• Researcher Bias: Researcher bias can affect the validity of a study. Biases can emerge through the researcher’s subjective interpretation, influence of personal beliefs, or preconceived notions, leading to unintentional distortion of findings or failure to consider alternative explanations.
• Ethical Validity: Limitations can arise if the study design or methods involve ethical concerns, such as the use of deceptive practices, inadequate informed consent, or potential harm to participants.

Also see  Reliability Vs Validity

About the author

Muhammad Hassan

Researcher, Academic Writer, Web developer

You may also like

Internal Consistency Reliability – Methods...

Internal Validity – Threats, Examples and Guide

Split-Half Reliability – Methods, Examples and...

Alternate Forms Reliability – Methods, Examples...

Reliability – Types, Examples and Guide

Test-Retest Reliability – Methods, Formula and...

Have a language expert improve your writing

Run a free plagiarism check in 10 minutes, generate accurate citations for free.

• Knowledge Base

Methodology

• The 4 Types of Validity in Research | Definitions & Examples

The 4 Types of Validity in Research | Definitions & Examples

Published on September 6, 2019 by Fiona Middleton . Revised on June 22, 2023.

Validity tells you how accurately a method measures something. If a method measures what it claims to measure, and the results closely correspond to real-world values, then it can be considered valid. There are four main types of validity:

• Construct validity : Does the test measure the concept that it’s intended to measure?
• Content validity : Is the test fully representative of what it aims to measure?
• Face validity : Does the content of the test appear to be suitable to its aims?
• Criterion validity : Do the results accurately measure the concrete outcome they are designed to measure?

In quantitative research , you have to consider the reliability and validity of your methods and measurements.

Note that this article deals with types of test validity, which determine the accuracy of the actual components of a measure. If you are doing experimental research, you also need to consider internal and external validity , which deal with the experimental design and the generalizability of results.

Table of contents

Construct validity, content validity, face validity, criterion validity, other interesting articles, frequently asked questions about types of validity.

Construct validity evaluates whether a measurement tool really represents the thing we are interested in measuring. It’s central to establishing the overall validity of a method.

What is a construct?

A construct refers to a concept or characteristic that can’t be directly observed, but can be measured by observing other indicators that are associated with it.

Constructs can be characteristics of individuals, such as intelligence, obesity, job satisfaction, or depression; they can also be broader concepts applied to organizations or social groups, such as gender equality, corporate social responsibility, or freedom of speech.

There is no objective, observable entity called “depression” that we can measure directly. But based on existing psychological research and theory, we can measure depression based on a collection of symptoms and indicators, such as low self-confidence and low energy levels.

What is construct validity?

Construct validity is about ensuring that the method of measurement matches the construct you want to measure. If you develop a questionnaire to diagnose depression, you need to know: does the questionnaire really measure the construct of depression? Or is it actually measuring the respondent’s mood, self-esteem, or some other construct?

To achieve construct validity, you have to ensure that your indicators and measurements are carefully developed based on relevant existing knowledge. The questionnaire must include only relevant questions that measure known indicators of depression.

The other types of validity described below can all be considered as forms of evidence for construct validity.

Receive feedback on language, structure, and formatting

Professional editors proofread and edit your paper by focusing on:

• Academic style
• Vague sentences
• Style consistency

See an example

Content validity assesses whether a test is representative of all aspects of the construct.

To produce valid results, the content of a test, survey or measurement method must cover all relevant parts of the subject it aims to measure. If some aspects are missing from the measurement (or if irrelevant aspects are included), the validity is threatened and the research is likely suffering from omitted variable bias .

A mathematics teacher develops an end-of-semester algebra test for her class. The test should cover every form of algebra that was taught in the class. If some types of algebra are left out, then the results may not be an accurate indication of students’ understanding of the subject. Similarly, if she includes questions that are not related to algebra, the results are no longer a valid measure of algebra knowledge.

Face validity considers how suitable the content of a test seems to be on the surface. It’s similar to content validity, but face validity is a more informal and subjective assessment.

You create a survey to measure the regularity of people’s dietary habits. You review the survey items, which ask questions about every meal of the day and snacks eaten in between for every day of the week. On its surface, the survey seems like a good representation of what you want to test, so you consider it to have high face validity.

As face validity is a subjective measure, it’s often considered the weakest form of validity. However, it can be useful in the initial stages of developing a method.

Criterion validity evaluates how well a test can predict a concrete outcome, or how well the results of your test approximate the results of another test.

What is a criterion variable?

A criterion variable is an established and effective measurement that is widely considered valid, sometimes referred to as a “gold standard” measurement. Criterion variables can be very difficult to find.

What is criterion validity?

To evaluate criterion validity, you calculate the correlation between the results of your measurement and the results of the criterion measurement. If there is a high correlation, this gives a good indication that your test is measuring what it intends to measure.

A university professor creates a new test to measure applicants’ English writing ability. To assess how well the test really does measure students’ writing ability, she finds an existing test that is considered a valid measurement of English writing ability, and compares the results when the same group of students take both tests. If the outcomes are very similar, the new test has high criterion validity.

If you want to know more about statistics , methodology , or research bias , make sure to check out some of our other articles with explanations and examples.

• Normal distribution
• Degrees of freedom
• Null hypothesis
• Discourse analysis
• Control groups
• Mixed methods research
• Non-probability sampling
• Quantitative research
• Ecological validity

Research bias

• Rosenthal effect
• Implicit bias
• Cognitive bias
• Selection bias
• Negativity bias
• Status quo bias

Face validity and content validity are similar in that they both evaluate how suitable the content of a test is. The difference is that face validity is subjective, and assesses content at surface level.

When a test has strong face validity, anyone would agree that the test’s questions appear to measure what they are intended to measure.

For example, looking at a 4th grade math test consisting of problems in which students have to add and multiply, most people would agree that it has strong face validity (i.e., it looks like a math test).

On the other hand, content validity evaluates how well a test represents all the aspects of a topic. Assessing content validity is more systematic and relies on expert evaluation. of each question, analyzing whether each one covers the aspects that the test was designed to cover.

A 4th grade math test would have high content validity if it covered all the skills taught in that grade. Experts(in this case, math teachers), would have to evaluate the content validity by comparing the test to the learning objectives.

Criterion validity evaluates how well a test measures the outcome it was designed to measure. An outcome can be, for example, the onset of a disease.

Criterion validity consists of two subtypes depending on the time at which the two measures (the criterion and your test) are obtained:

• Concurrent validity is a validation strategy where the the scores of a test and the criterion are obtained at the same time .
• Predictive validity is a validation strategy where the criterion variables are measured after the scores of the test.

Convergent validity and discriminant validity are both subtypes of construct validity . Together, they help you evaluate whether a test measures the concept it was designed to measure.

• Convergent validity indicates whether a test that is designed to measure a particular construct correlates with other tests that assess the same or similar construct.
• Discriminant validity indicates whether two tests that should not be highly related to each other are indeed not related. This type of validity is also called divergent validity .

You need to assess both in order to demonstrate construct validity. Neither one alone is sufficient for establishing construct validity.

The purpose of theory-testing mode is to find evidence in order to disprove, refine, or support a theory. As such, generalizability is not the aim of theory-testing mode.

Due to this, the priority of researchers in theory-testing mode is to eliminate alternative causes for relationships between variables . In other words, they prioritize internal validity over external validity , including ecological validity .

It’s often best to ask a variety of people to review your measurements. You can ask experts, such as other researchers, or laypeople, such as potential participants, to judge the face validity of tests.

While experts have a deep understanding of research methods , the people you’re studying can provide you with valuable insights you may have missed otherwise.

Cite this Scribbr article

If you want to cite this source, you can copy and paste the citation or click the “Cite this Scribbr article” button to automatically add the citation to our free Citation Generator.

Middleton, F. (2023, June 22). The 4 Types of Validity in Research | Definitions & Examples. Scribbr. Retrieved March 18, 2024, from https://www.scribbr.com/methodology/types-of-validity/

Is this article helpful?

Fiona Middleton

Other students also liked, reliability vs. validity in research | difference, types and examples, construct validity | definition, types, & examples, external validity | definition, types, threats & examples, what is your plagiarism score.

The Quintessence of Basic and Clinical Research and Scientific Publishing pp 769–781 Cite as

Writing a Postgraduate or Doctoral Thesis: A Step-by-Step Approach

• Usha Y. Nayak 4 ,
• Praveen Hoogar 5 ,
• Srinivas Mutalik 4 &
• N. Udupa 6  
• First Online: 01 October 2023

581 Accesses

1 Citations

A key characteristic looked after by postgraduate or doctoral students is how they communicate and defend their knowledge. Many candidates believe that there is insufficient instruction on constructing strong arguments. The thesis writing procedure must be meticulously followed to achieve outstanding results. It should be well organized, simple to read, and provide detailed explanations of the core research concepts. Each section in a thesis should be carefully written to make sure that it transitions logically from one to the next in a smooth way and is free of any unclear, cluttered, or redundant elements that make it difficult for the reader to understand what is being tried to convey. In this regard, students must acquire the information and skills to successfully create a strong and effective thesis. A step-by-step description of the thesis/dissertation writing process is provided in this chapter.

• Dissertation
• Postgraduate
• SMART objectives

This is a preview of subscription content, log in via an institution .

Buying options

• Available as PDF
• Read on any device
• Instant download
• Own it forever
• Available as EPUB and PDF
• Durable hardcover edition
• Dispatched in 3 to 5 business days
• Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Carter S, Guerin C, Aitchison C (2020) Doctoral writing: practices, processes and pleasures. Springer, Singapore. https://doi.org/10.1007/978-981-15-1808-9

Book   Google Scholar  

Odena O, Burgess H (2017) How doctoral students and graduates describe facilitating experiences and strategies for their thesis writing learning process: a qualitative approach. Stud High Educ 42:572–590. https://doi.org/10.1080/03075079.2015.1063598

Article   Google Scholar  

Stefan R (2022) How to write a good PhD thesis and survive the viva, pp 1–33. http://people.kmi.open.ac.uk/stefan/thesis-writing.pdf

Google Scholar  

Barrett D, Rodriguez A, Smith J (2021) Producing a successful PhD thesis. Evid Based Nurs 24:1–2. https://doi.org/10.1136/ebnurs-2020-103376

Article   PubMed   Google Scholar  

Murray R, Newton M (2009) Writing retreat as structured intervention: margin or mainstream? High Educ Res Dev 28:541–553. https://doi.org/10.1080/07294360903154126

Thompson P (2012) Thesis and dissertation writing. In: Paltridge B, Starfield S (eds) The handbook of english for specific purposes. John Wiley & Sons, Ltd, Hoboken, NJ, pp 283–299. https://doi.org/10.1002/9781118339855.ch15

Chapter   Google Scholar  

Faryadi Q (2018) PhD thesis writing process: a systematic approach—how to write your introduction. Creat Educ 09:2534–2545. https://doi.org/10.4236/ce.2018.915192

Faryadi Q (2019) PhD thesis writing process: a systematic approach—how to write your methodology, results and conclusion. Creat Educ 10:766–783. https://doi.org/10.4236/ce.2019.104057

Fisher CM, Colin M, Buglear J (2010) Researching and writing a dissertation: an essential guide for business students, 3rd edn. Financial Times/Prentice Hall, Harlow, pp 133–164

Ahmad HR (2016) How to write a doctoral thesis. Pak J Med Sci 32:270–273. https://doi.org/10.12669/pjms.322.10181

Article   CAS   PubMed   PubMed Central   Google Scholar  

Gosling P, Noordam LD (2011) Mastering your PhD, 2nd edn. Springer, Berlin, Heidelberg, pp 12–13. https://doi.org/10.1007/978-3-642-15847-6

Cunningham SJ (2004) How to write a thesis. J Orthod 31:144–148. https://doi.org/10.1179/146531204225020445

Article   CAS   PubMed   Google Scholar  

Azadeh F, Vaez R (2013) The accuracy of references in PhD theses: a case study. Health Info Libr J 30:232–240. https://doi.org/10.1111/hir.12026

Williams RB (2011) Citation systems in the biosciences: a history, classification and descriptive terminology. J Doc 67:995–1014. https://doi.org/10.1108/00220411111183564

Bahadoran Z, Mirmiran P, Kashfi K, Ghasemi A (2020) The principles of biomedical scientific writing: citation. Int J Endocrinol Metab 18:e102622. https://doi.org/10.5812/ijem.102622

Article   PubMed   PubMed Central   Google Scholar  

Yaseen NY, Salman HD (2013) Writing scientific thesis/dissertation in biology field: knowledge in reference style writing. Iraqi J Cancer Med Genet 6:5–12

Gorraiz J, Melero-Fuentes D, Gumpenberger C, Valderrama-Zurián J-C (2016) Availability of digital object identifiers (DOIs) in web of science and scopus. J Informet 10:98–109. https://doi.org/10.1016/j.joi.2015.11.008

Khedmatgozar HR, Alipour-Hafezi M, Hanafizadeh P (2015) Digital identifier systems: comparative evaluation. Iran J Inf Process Manag 30:529–552

Kaur S, Dhindsa KS (2017) Comparative study of citation and reference management tools: mendeley, zotero and read cube. In: Sheikh R, Mishra DKJS (eds) Proceeding of 2016 International conference on ICT in business industry & government (ICTBIG). Institute of Electrical and Electronics Engineers, Piscataway, NJ. https://doi.org/10.1109/ICTBIG.2016.7892715

Kratochvíl J (2017) Comparison of the accuracy of bibliographical references generated for medical citation styles by endnote, mendeley, refworks and zotero. J Acad Librariansh 43:57–66. https://doi.org/10.1016/j.acalib.2016.09.001

Zhang Y (2012) Comparison of select reference management tools. Med Ref Serv Q 31:45–60. https://doi.org/10.1080/02763869.2012.641841

Hupe M (2019) EndNote X9. J Electron Resour Med Libr 16:117–119. https://doi.org/10.1080/15424065.2019.1691963

Download references

Author information

Authors and affiliations.

Department of Pharmaceutics, Manipal College of Pharmaceutical Sciences, Manipal Academy of Higher Education, Manipal, Karnataka, India

Usha Y. Nayak & Srinivas Mutalik

Centre for Bio Cultural Studies, Directorate of Research, Manipal Academy of Higher Education, Manipal, Karnataka, India

Praveen Hoogar

Shri Dharmasthala Manjunatheshwara University, Dharwad, Karnataka, India

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to N. Udupa .

Editor information

Editors and affiliations.

Retired Senior Expert Pharmacologist at the Office of Cardiology, Hematology, Endocrinology, and Nephrology, Center for Drug Evaluation and Research, US Food and Drug Administration, Silver Spring, MD, USA

Gowraganahalli Jagadeesh

Professor & Director, Research Training and Publications, The Office of Research and Development, Periyar Maniammai Institute of Science & Technology (Deemed to be University), Vallam, Tamil Nadu, India

Pitchai Balakumar

Division Cardiology & Nephrology, Office of Cardiology, Hematology, Endocrinology and Nephrology, Center for Drug Evaluation and Research, US Food and Drug Administration, Silver Spring, MD, USA

Fortunato Senatore

Ethics declarations

No conflict of interest exists.

Rights and permissions

Reprints and permissions

Copyright information

© 2023 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.

About this chapter

Cite this chapter.

Nayak, U.Y., Hoogar, P., Mutalik, S., Udupa, N. (2023). Writing a Postgraduate or Doctoral Thesis: A Step-by-Step Approach. In: Jagadeesh, G., Balakumar, P., Senatore, F. (eds) The Quintessence of Basic and Clinical Research and Scientific Publishing. Springer, Singapore. https://doi.org/10.1007/978-981-99-1284-1_48

Download citation

DOI : https://doi.org/10.1007/978-981-99-1284-1_48

Published : 01 October 2023

Publisher Name : Springer, Singapore

Print ISBN : 978-981-99-1283-4

Online ISBN : 978-981-99-1284-1

eBook Packages : Biomedical and Life Sciences Biomedical and Life Sciences (R0)

Share this chapter

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Publish with us

Policies and ethics

• Find a journal
• Track your research

Want to Get your Dissertation Accepted?

Discover how we've helped doctoral students complete their dissertations and advance their academic careers!

Join 200+ Graduated Students

Get Your Dissertation Accepted On Your Next Submission

Get customized coaching for:.

• Crafting your proposal,
• Collecting and analyzing your data, or
• Preparing your defense.

Trapped in dissertation revisions?

External validity: everything you need to know, published by steve tippins on december 18, 2020 december 18, 2020.

Last Updated on: 29th August 2022, 08:16 am

Everybody knows external validity is important. With it, you’ll soar to the heights of research applicability. Without it, you’ll barely make it off the ground. But… what exactly is it? 

Fear not: if you’re confused about what external validity is and how to achieve it, you’re in the right place. Or, if you already know what it is but haven’t the slightest clue how to actually put it into practice, we’ve got you covered too.

What Is External Validity?

When we consider external validity, we are asking whether the results of our study can be generalized beyond the scope of the study itself. Can the claim be realistically applied to larger populations, as well as to other times or situations? 

When and How Is External Validity Important?

External validity is extremely important with frequency claims — studies that conclude how frequent or common something is. For example, “14% of College Students Consider Suicide” is a frequency claim.  For us to take this claim seriously, we would need to know how they chose their study participants — did they ask a few students on the sidewalk? Did they ask 100 randomly chosen students across a variety of colleges?

Association claims — studies that claim that two things often occur together — also require interrogation of external validity.   For example, “People Who Talk with Their Hands Are Often Warmer and Friendlier Than Those Who Don’t” is an example of an association claim.  Do these results always hold true?  Can a study of middle school girls in Connecticut that showed these results also hold true for a group of middle-aged men in California?

When saying one variable causes another (causal claims), we ask: to what populations, settings, and times can we generalize?  For a study that claims that music lessons increase IQ, we would need to ask whether the results would apply in all cultures, for people of all socio-economic backgrounds, and at all ages. When making causal claims, however, interrogation is usually more rigorously focused on internal validity.

What’s the Difference Between External and Internal Validity? 

Internal validity refers to the construction of the study and means that conclusions are warranted, extraneous variables are controlled, alternative explanations are eliminated, and accurate research methods were used. 

External validity means the degree to which findings can be generalized beyond the sample, the outcomes apply to practical situations, and that the results can be translated into another context.

An Oregon State University researcher talks about the difference in practice:

In some more recent work, I was looking specifically at physical activity during pregnancy as the only exposure; thus, my advertisement to recruit women into the study mentioned that I was studying exercise in pregnancy (rather than pregnancy in general). [1] In this more recent study, I had very few sedentary people—indeed, I have a few who reported running half marathons while pregnant! Since this is not normal, my study—though it does have reasonable internal validity—cannot be generalized to all pregnant women but only to the subpopulation of them who get a fair bit of physical activity. It lacks external validity.  Because it has good internal validity, I can generalize the results to highly active pregnant women—just not to all pregnant women.

Threats to External Validity

In order for your research to apply to other contexts (and thus be of use to anyone), it’s important to manage or eliminate threats to external validity. Here are some of the most common threats.

Sampling Errors 

Polling during political campaign season offers a vivid example of the importance of sampling for predicting the behavior of a population: “Can we predict the results of the presidential election based on this sample of 1200 people?” 

You’ll want to know if the sample comes from the population of interest , so you’ll want to define that first.  Pollsters don’t ask children who they’d vote for, because children are not allowed to vote — they are not representative of the population of interest.

You’ll also need to be sure that the sample is representative of the population .  Polls that sample only adult white males would be unlikely to accurately predict the election because they do not adequately represent the diversity of the voting population.

Special Circumstances

You also want to make sure major historical events do not influence the results of your study. For example, doing your study during a global pandemic. Does the pandemic have any influence on your results?

Or, if you planned to survey people over a time period (say, three months) and two months into the process, there’s a big change in society or within the profession, that might make your results have less external validity. Because some of the responses came before the major event and some came after, it’s hard to tell the impact the event had on the results.

Hack Your Dissertation

5-Day Mini Course: How to Finish Faster With Less Stress

Interested in more helpful tips about improving your dissertation experience? Join our 5-day mini course by email!

How to Achieve External Validity

Larger, representative sample.

Sample size is always a trade-off. Dedicating time and money needed to accumulate a large sample increases external validity, but a smaller sample allows for faster completion with fewer resources. Most people do a G*Power test, when doing a quantitative study, to determine the minimum sample size that’s needed to allow for external validity.  

You want the sample to be large enough to sufficiently limit the influence of outliers.   In a larger sample, someone offering unrepresentative views/behavior will not skew the results. For example, in a study where 4 people are asked how many doughnuts they can eat and 3 eat 2 doughnuts and 1 eats 34, the average is 10 doughnuts per person. However, if 1,000 people are surveyed and 999 eat 2 doughnuts and 1 eats 34 the average is 2.032. The outlier does not skew the results much. 

Random samples are considered more valid than purposive samples, because you have a better chance of representing the population randomly than if you select who will be part of the study, or if they volunteer.  

Replicability

If your study is set up so that others can repeat your study, then results have the potential to become much more externally valid. If others repeat the study and come up with similar results, that means your findings might actually be the case generally. If you replicate your own study under different circumstances or with different populations, you have more results to make your conclusion stronger. 

Final Thoughts

External validity is an important concept to understand. A lack of understanding may doom your study, while strong external validity makes getting your dissertation accepted and/or article much easier to get published.

Steve Tippins

Steve Tippins, PhD, has thrived in academia for over thirty years. He continues to love teaching in addition to coaching recent PhD graduates as well as students writing their dissertations. Learn more about his dissertation coaching and career coaching services. Book a Free Consultation with Steve Tippins

Related Posts

Dissertation

What makes a good research question.

Creating a good research question is vital to successfully completing your dissertation. Here are some tips that will help you formulate a good research question.  What Makes a Good Research Question? These are the three Read more…

Dissertation Structure

When it comes to writing a dissertation, one of the most fraught questions asked by graduate students is about dissertation structure. A dissertation is the lengthiest writing project that many graduate students ever undertake, and Read more…

Choosing a Dissertation Chair

Choosing your dissertation chair is one of the most important decisions that you’ll make in graduate school. Your dissertation chair will in many ways shape your experience as you undergo the most rigorous intellectual challenge Read more…

Make This Your Last Round of Dissertation Revision.

Learn How to Get Your Dissertation Accepted .

Discover the 5-Step Process in this Free Webinar .

Almost there!

Please verify your email address by clicking the link in the email message we just sent to your address.

If you don't see the message within the next five minutes, be sure to check your spam folder :).

• Open access
• Published: 07 September 2023

Validity, acceptability, and procedural issues of selection methods for graduate study admissions in the fields of science, technology, engineering, and mathematics: a mapping review

• Anastasia Kurysheva   ORCID: orcid.org/0000-0001-7425-1345 1 , 2 ,
• Harold V. M. van Rijen 1 , 2 ,
• Cecily Stolte 1 , 2 &
• Gönül Dilaver   ORCID: orcid.org/0000-0002-6227-2197 1 , 2  

International Journal of STEM Education volume  10 , Article number:  55 ( 2023 ) Cite this article

2128 Accesses

4 Altmetric

Metrics details

This review presents the first comprehensive synthesis of available research on selection methods for STEM graduate study admissions. Ten categories of graduate selection methods emerged. Each category was critically appraised against the following evaluative quality principles: predictive validity and reliability, acceptability, procedural issues, and cost-effectiveness. The findings advance the field of graduate selective admissions by (a) detecting selection methods and study success dimensions that are specific for STEM admissions, (b) including research evidence both on cognitive and noncognitive selection methods, and (c) showing the importance of accounting for all four evaluative quality principles in practice. Overall, this synthesis allows admissions committees to choose which selection methods to use and which essential aspects of their implementation to account for.

Introduction

A high-quality student selection procedure for graduate level education is of utmost importance for programs, students, and society. Higher education has seen several influential policy developments over the past decades such as the introduction of the Bologna Process in 1999 in Europe and the increased internationalization of higher education across the globe (De Wit & Altbach, 2020 ). These policies contributed to rising international/cross-border and national (i.e., between higher education institutions within one country) student mobility (Okahana & Zhou, 2018 ; Payne, 2015 ). The knock-on effect of this mobility has created a growing diversity of graduate application files. Admissions committees are now faced with applicants from different higher education systems, potentially a variety of background fields, and varying levels of academic skills and proficiency in the language of instruction.

Furthermore, the problem of underrepresentation of students with certain backgrounds persists across the globe, including countries with well-developed higher education systems (Salmi & Bassett, 2014 ). As such, it is still more difficult for students with low socioeconomic status (SES), a migration background, those who are first-generation students, or students with disabilities to gain admissions into higher education programs compared to students with middle/high SES, no migration background, parents who hold academic degree, or students without disabilities (Garaz & Torotcoi, 2017 ; Salmi & Bassett, 2014 ; Weedon, 2017 ). Students’ application files are often conditioned by their background: For example, students with parents of low SES cannot typically show an impressive list of extracurricular activities on their resume in contrast to their peers with parents of high SES (Jayakumar & Page, 2021 ). Since these factors contribute to the inequality already at the entrance to higher education—at the undergraduate level (Zimdars, 2016 ), they may further exacerbate their effects in the selective graduate level of education, where there are even fewer places available. It is, therefore, often the case that a straightforward assessment of application files is not feasible because of the multifaceted nature of each application. Unsurprisingly, it is a complex task for admissions committees to evaluate the educational background and achievements of (inter)national students with diverse backgrounds. Regardless of described complexities, admissions decisions must be objective, fair, and transparent to ensure their adequate justification.

Evaluative quality principles

To facilitate the achievement of the overarching goals of objectivity, fairness, and transparency, four evaluative quality principles regarding student selection methods were recognized as essential (Patterson et al., 2016 ):

Effectiveness combines both (predictive) (incremental) validity and reliability. This principle encompasses several questions that should ideally be considered together: Does a selection method predict study success and to what extent? Even if a selection method does predict study success, does it provide additional value beyond other valid selection methods? Does the use of a selection method deliver consistent results across time, locations, and assessors?

Procedural issues of a selection method refer to any aspects that are important in the practical implementation of the method such as its limitations, the impact of its structure and format on its effectiveness, any biases that are naturally integrated into its design etc.

Acceptability refers to both the willingness to implement a selection method and the satisfaction of stakeholders from its usage. Relevant questions in this regard are: How widely is the selection method used across different disciplines, countries, and regions? To what extent are admissions committees willing to apply the method? Do they find it useful? Finally, how much do applicants favor the selection method?

Cost-effectiveness is a quality evaluative principle that refers to the financial impact of a selection method on educational programs and applicants. In other words, it refers to the questions: Who pays for its usage in the admissions process, and how much does it cost?

There is a striking lack of studies that synthesize research evidence on selection methods for graduate study admissions while accounting for all four evaluative quality principles. Instead, the existing reviews and meta-analyses address evidence for each selection method separately: standardized testing (Kuncel & Hezlett, 2007b , 2010 ; Kuncel et al., 2004 , 2010 ), recommendation letters (Kuncel et al., 2014 ), personal statements (Murphy et al., 2009 ), and other various noncognitive measures (Kuncel et al., 2020 ; Kyllonen et al., 2005 , 2011 ; Megginson, 2009 ). Moreover, these studies usually focus on predictive validity and rarely on procedural issues, with only limited or no attention to reliability, acceptability, and cost-effectiveness.

The only review to combine evidence on all available selection methods within one study and included the four evaluative quality principles (validity/reliability, procedural issues, acceptability, and cost-effectiveness) was conducted by Patterson et al. ( 2016 ). However, this review only focused on selection methods in medical education. For example, it does not present evidence on (nonmedical) standardized tests of academic aptitude, tests of language of instruction, or amount and quality of prior research experience. Therefore, its findings can only be partially generalized for graduate admissions.

The question that arises is which educational field (except medical education) has attracted enough high-quality research that (a) addresses the four evaluative quality principles and (b) allows admissions committees to use the findings in a wide range of graduate programs, therefore, enhancing the potential impact of this review? From the preliminary overview, we think that science, technology, engineering, and mathematics (STEM) fields meet these two conditions. STEM fields have been recognized worldwide as fundamental for finding solutions to urgent societal problems (Proudfoot & Hoffer, 2016 ). The efforts of certain countries to become leaders in STEM higher education and research (e.g., China; Kirby & van der Wende, 2019 ) are illustrative of how crucial the STEM fields are for economic growth and prosperity. Unsurprisingly, STEM disciplines have attracted a rising number of students, making research evidence on selection methods for STEM studies increasingly more relevant. Since there has been no synthesis of such evidence to date, we designed this review to address this gap.

The present review

The aim of this review is to present a comprehensive overview of research evidence on the existing selection methods in graduate admissions in STEM fields. The review focuses on evaluative quality principles of validity, reliability, procedural issues, acceptability, and cost-effectiveness. The term “graduate” refers to both master’s and doctoral levels. That is, studies on both levels were collected for this review.

Research questions

What evidence is provided in research literature within STEM graduate admissions field on:

the extent to which different selection methods are valid and reliable?

procedural issues of the selection methods?

the extent to which different selection methods are accepted by stakeholders?

the extent to which different selection methods are cost-effective?

For this review, a systematic search was conducted and complemented with an expanded search of literature in reference lists of relevant books and articles.

Inclusion criteria for the literature review

The inclusion criteria for this review were: (1) the topic on selection methods in graduate admissions, (2) the graduate level of education (i.e., master’s and/or PhD phase), (3) samples that include students from STEM disciplines, (4) studies addressing at least one of four evaluative quality principles of interest: validity/reliability, procedural issues, acceptability, and cost-effectiveness, (5) studies conducted in at least one of the Organization for Economic Co-operation and Development (OECD) countries, (6) studies published in English, (7) studies that went through a peer-review process, (8) studies conducted in the period between 2005 and June 2023.

The OECD countries were chosen because of their well-developed higher education systems as well as an expectation that the quality of research in these countries is comparable. The time frame was chosen in accordance with the changes in European higher education systems after the introduction of the Bologna Process (The Bologna Declaration, 1999 ). Countries joined the process in different subsequent years. Therefore, 2005 was chosen as a plausible cut-off moment to account for the fact that the first students, studying within the new system, could graduate. The same time frame was applied for the US research context.

We chose to review the literature, referring to master’s and PhD levels together (that is, on a graduate level overall), because the training on both levels is advanced. Furthermore, many studies that were included in this review did not make a distinction between the two levels. We also considered different STEM majors or contexts (e.g., the European vs. the US contexts) together, because we aimed to detect overarching patterns in evaluative quality principles that would be applicable to a variety of majors and higher education contexts on a graduate level.

The literature search procedure

The literature search delivered 3244 potentially relevant items including duplicates. The main portion of the results was obtained via conducting a systematic search in the specialized databases (ERIC: n  = 1089; PsycInfo: n  = 1112; Medline: n  = 234; Scopus: n  = 649). The keywords of the systematic search can be found in Additional file 1 : Table S1. The syntax for each database is available upon request. While we did not have the opportunity to carry out searches in all specific databases for each STEM education field (e.g., databases focusing on engineering education), we expect that the large educational data bases such as ERIC contain a substantial number of studies related to our topic in each of those fields. Next, the literature search was extended beyond the database approaches. Namely, the citations from relevant articles were examined ( n  = 71), and previously collected research literature was added ( n  = 89). The screening was conducted in two steps. In the first step, the titles and abstracts were scanned to remove duplicates and obviously irrelevant search results. In the second step, the full texts of remaining articles were obtained and examined. The full texts of four articles were not found even after contacting the authors and were not included in the final number.

Figure  1 presents a detailed flowchart of the steps undertaken. Two coders (the first and the third authors) conducted both steps of screenings. To ensure that the same papers were selected, both coders screened all papers at both steps according to the inclusion criteria. They used codes, such as “yes”, “no”, and “may be”, with the later meaning that an article required a joint decision during the discussion. All papers were independently screened by the two coders during both steps. Although the agreement after the first screening was near complete (kappa = 0.88) and that of the second screening was strong (kappa = 0.70), there were papers with different codes (e.g., “yes” and “may be”, or more rarely “yes” and “no”) or about which the coders had doubts (a code of “may be”). All such disagreements were resolved through discussion.

Flowchart of articles’ selection

In total, 77 articles met the inclusion criteria for this review. The distribution across the OECD countries is presented in Table 1 . The distribution across STEM disciplines is presented in Table 2 .

After the screening was completed, the graduate selection methods from 77 studies were assigned into ten categories: (1) prior grades, (2) standardized testing of academic abilities, (3) letters of recommendation, (4) interviews, (5) personal statements (i.e., motivation letters), (6) personality assessments, (7) intelligence assessments, (8) language proficiency, (9) prior research experience, and (10) various, rarely studied selection methods that do not fall under more common methods above (such as resumes, selectivity of prior higher education institution (HEI), former (type of) HEI, amount and quality of research experience, or composite scores). If one study addressed different methods or evaluative quality principles, that study was included in all respective categories. The numbers of papers cross-tabulated according to selection method and evaluative quality principle are presented in Additional file 1 : Table S2. Additional file 1 : Table S3 shows the main characteristics of studies, such as study design, country, field of study, and so forth. Additional file 1 : Table S3 also includes the summary of the relevant findings per study.

Contributions of this review

The main contribution of this review is that it synthesizes high-quality research evidence across four evaluative quality principles, as proposed by Patterson et al. ( 2016 ), for both cognitive and noncognitive selection methods. No such synthesis has been conducted in the field of STEM graduate admissions (For an overview of the assessment of only noncognitive constructs in graduate education, one may consult the papers of de Boer and Van Rijnsoever, 2022a ; Kyllonen et al., 2005 , 2011 ). Another strong aspect of this review is that it compares the findings of primary and secondary (i.e., reviews, meta-analyses) studies, wherever possible. This is important considering possible limitations of primary studies, such as range restriction and criteria unreliability, which can be accounted for in meta-analyses (Sedlacek, 2003 ). Overall, this review aims to provide a compilation of state-of-the-art research on selective graduate admissions in STEM fields of study.

Additional file 1 : Table S2 shows the numbers of articles on each selection method and evaluative quality principle. We note the overall lack of research on the topics of reliability and cost-effectiveness. Therefore, the evidence below is presented mostly on validity, acceptability, and procedural issues. When studies on reliability or cost-effectiveness are available, they are reported in the respective selection methods’ categories.

Prior grades

Validity and reliability of prior grades.

The research focused on exploring the predictive validity of different aspects of grade point average (GPA), such as undergraduate GPA (UGPA), the first-year GPA, and the last-year GPA. Findings and relevant references are presented in Table 3 . Overall, it appears that UGPA is a valid predictor of graduate degree completion, student performance on introductory graduate courses, and graduate GPA (GGPA). However, UGPA is not valid for predicting research productivity (defined as number of published papers, presentations, and obtained grants) and passing qualifying exams. There is mixed evidence on predictive validity of UGPA toward time to graduate degree and faculty ratings.

Some single studies looked at UGPA in more detail. Namely, they disentangled UGPA on first-year UGPA and last-year GPA. A study that tried to predict graduate degree completion with first-year UGPA found no such relationship (DeClou, 2016 ). A study that explored the predictive validity of last-year UGPA found that last-year GPA is positively related to GGPA (Zimmermann et al., 2017a ).

We found one study that addressed the question of reliability estimates. The author calculated eight different reliability coefficients for fourth-year cumulative GPA at each higher education institution included in the study and then meta-analyzed them (Westrick, 2017 ). The study showed that the various reliability estimates ranged between 0.89 and 0.92. The author recommends using stratified alpha as a reliability coefficient for cumulative GPA, which works best with the multi-factor data, due to the variation in the processes involved in earning grades in the first-year and fourth-year courses (Westrick, 2017 ).

Procedural issues of prior grades

There are several procedural issues with using prior grades for admissions decisions. The first one is grade inflation—a practice of awarding higher grades than previously assigned for given levels of achievement (Merriam-Webster dictionary, n.d. ): For example, teachers giving higher grades for positive student ratings (European Grade Conversion System [EGRACONS], 2020 ). In her observational study of top graduate research programs, Posselt ( 2014 ) indicated that grade inflation is a widespread phenomenon in highly selective universities. In such universities, students from underrepresented backgrounds are extremely lacking; therefore, setting a grade-threshold on a high level disproportionately excluded these students (Posselt, 2014 ).

The second one refers to differences in grading standards, which relates to the fact that one grade obtained at different institutions might reflect a different level of academic qualification. Grade conversion and grade distribution tables, which are developed to tackle these issues, are not without limitations. They can often be crude, and this can affect both selection decisions and research done on grades as predictors of graduate study success (see, e.g., Zimmermann et al., 2017a ).

The third procedural issue relates to a possibility of cognitive biases of assessors to influence grading: This could be an origin of differences in prior grades observed between applicants with various socioeconomic status (SES), genders, and races (Woo et al., 2023 ). Finally, the relatedness, or fit, between undergraduate and graduate programs affects the predictive value of grades received during undergraduate studies: When the programs are related to a high extent, the relationship between undergraduate and graduate grades is stronger compared to a situation when the undergraduate and graduate programs are related to a low extent (de Boer & Rijnsoever, 2022b ).

Acceptability of prior grades

Prior grades are a widely accepted selective admissions method (Boyette-Davis, 2018 ; MasterMind Europe, 2017 ). The largest weight in admissions decisions is given to grades on undergraduate courses that are closest in terms of content to the courses of a graduate program (Chari & Potvin, 2019 ). When explaining what the reasons are behind high acceptability of grades and even overestimation of their importance in graduate admissions by admissions committees, Posselt ( 2014 ) states that high conventual achievements, such as grades, are consistent with the identity of an elite intellectual community, which admissions committee members, implicitly or explicitly, refer themselves.

Standardized testing of academic abilities

Validity of standardized admissions tests of academic abilities.

Among different standardized admissions tests, the ones which are typically required for selective admissions to graduate programs in STEM disciplines are the Graduate Record Examinations (GRE) General and GRE Subject. All but one study, which addressed validity of standardized tests, referred to these two GRE tests. The only exception was the standardized test EXANI-III, which is used in Mexico.

Validity of graduate standardized admissions tests has been a controversial topic in research, with some studies providing evidence for their weak-to-moderate predictive power toward graduate study success and others indicating the absence of predictive power (see Table 4 ). From Table 4 , we can infer that the standardized test most often examined is the GRE General.

The GRE General is a positive predictor of first-year GGPA, GGPA, and faculty ratings. This is in line with the existing reviews and meta-analyses (Kuncel & Hezlett, 2007b , 2010 ; Kuncel et al., 2010 ). From the majority of primary studies, it appears that the GRE General does not predict graduate degree completion and research productivity defined as the number of publications.

The meta-analyses on the topic, however, found that after meta-analytical corrections for statistical artifacts in primary studies were applied (such as a correction for the restriction of range of a predictor), these two relationships (1) between the GRE General and degree completion and (2) between the GRE General and research productivity, although weak, were detected (Kuncel & Hezlett, 2007a , 2007b ).

Finally, there was mixed or limited evidence for GRE General efficiency in prediction of time to graduate degree, performance on core program courses, qualifying exam, rate of progress, and thesis performance (see Table 4 for details).

There is an indication that another standardized test, the GRE Subject in Physics, is predictive for faculty ratings, while its predictive value for graduate degree completion remains unclear. Two meta-analyses also found that the GRE Subject is a meaningful predictor of graduate study success (Kuncel & Hezlett, 2007b ; Kuncel et al., 2010 ).

Procedural issues of standardized admissions tests of academic abilities

The primary studies showed a possibility of (1) adverse impact of the GRE on underrepresented groups (including ethnic minorities and females in STEM), which can be mitigated by applying a systematic and holistic approach in reviewing admissions files (Bleske-Rechek & Browne, 2014 ; Murphy, 2009 ; Posselt, 2014 ; Wilson et al., 2018 , 2019 ), and (2) item position effects, which can be mitigated by allowing proper time limits for taking the test (Davey & Lee, 2011 ).

However, the reviews and meta-analyses on procedural issues refuted several common beliefs regarding standardized tests, such as: (1) the coaching effects, which were shown to be modest with one quarter of a standard deviation improvement in test performance (Hausknecht et al., 2007 ; Kuncel & Hezlett, 2007a , 2007b ). Such an improvement refers primarily toward the GRE Analytical Writing section (GRE-A) (Powers, 2017 ). GRE Verbal Reasoning (GRE-V) and GRE Quantitative Reasoning (GRE-Q) were prone to coaching to a negligible extent in contrast to claims of commercial organizations that prepare test takers for standardized tests (Powers, 2017 ) ; (2) lack of predictive independence from SES, which was contested by demonstrating that even after controlling for SES, standardized test scores remained predictive of study success (Camara et al., 2013 ; Kuncel & Hezlett, 2010 ); (3) bias in testing. Some researchers state that bias in graduate testing is a myth, as, according to their findings, standardized tests appeared to predict graduate study success of both females and males equally (Fischer et al., 2013 ; Kuncel & Hezlett, 2007b ) as well as ethnic groups (Kuncel & Hezlett, 2007b ). The authors of these studies also indicated that the differences in performance between different groups might reflect societal problems, such as lack of family, social, environmental, peer, and financial support. They state that standardized tests simply expose the preexisting differences created by the above-mentioned societal problems (Camara et al., 2013 ; Kuncel & Hezlett, 2010 ); (4) negative effect of stereotype threat on standardized test performance: Test takers, who believe that their nonoptimal performance on standardized tests might confirm the stereotypes of their minority group’s intellectual capacity, might perform worse because of that self-fulfilling prophecy (Garces, 2014 ).

Acceptability of standardized admissions tests of academic abilities

Acceptability by admissions committees In the US context, admissions committees—especially for research programs—actively use the GRE General and consider it to be a valuable contributor for their admissions decisions (Boyette-Davis, 2018 ; Chari & Potvin, 2019 ; Rock & Adler, 2014 ). Out of the three sections, GRE-V and GRE-Q are used most, while GRE-A is considered the least often (only around 35% of surveyed programs; Briihl & Wasieleski, 2007 ). When it comes to positioning GRE as a selection method, the GRE appeared less important than, for example, previous research experience, UGPA, and certain personal characteristics (e.g., critical thinking, work ethics; Boyette-Davis, 2018 ). However, the GRE had more weight in selection decisions for doctoral programs than for masters’ programs (Chari & Potvin, 2019 ).

A survey among masters’ programs in Europe showed that the results of standardized admissions tests are rarely used for elimination purposes (only around 5% masters’ programs admitted such a practice), but higher scores, if present, do provide an advantage to students in one fourth of the programs (MasterMind Europe, 2017 ). However, Europe has seen a steady increase in GRE test takers (e.g., it increased from 12,243 in 2004 to 29,211 in 2013) since the introduction of the Bologna Process and the increasing internationalization of European graduate education (Payne, 2015 ). Test takers aiming to study STEM disciplines represented the largest group among all European GRE test takers (Payne, 2015 ).

Acceptability by applicants Applicants viewed the GRE as less important in graduate admissions than UGPA, recommendation letters, and work experience (Cline & Powers, 2014 ). Applicants coming from racial minority groups had more negative feelings about the GRE than white test takers (Cline & Powers, 2014 ). International students felt that the GRE is culturally biased (Mupinga & Mupinga, 2005 ). Applicants perceived publishing prompts from GRE-A positively (Powers, 2005 ) and desired to get additional information about their writing skills beyond their GRE-A score (Attali & Sinharay, 2015 ).

Cost-effectiveness of standardized admissions tests of academic abilities

One study looked at this evaluative quality principle. In their study, Klieger et al. ( 2014 ) provided an example of calculation of the benefits for one US doctoral program. They estimated the financial benefits of using the GRE for admissions and funding decisions as considerable, but obviously, the exact numbers will depend on a specific program and a number of GRE sections used for admissions decisions.

Letters of recommendation (LoRs)

Validity and reliability of letters of recommendation.

The only primary study which examined predictive validity of LoRs for STEM disciplines (namely, the biomedical sciences) found that the scores on LoRs did not predict time to degree, but they were the most powerful predictor of first-author student publications (Hall et al., 2017 ). The review of Kuncel et al. ( 2014 ) showed that LoRs do not deliver incremental validity over standardized admissions tests and UGPA toward GGPA and faculty ratings but do deliver small incremental validity in prediction of degree completion (an outcome usually difficult to predict using other measures). The review of Megginson ( 2009 ) showed that narrative LoRs have minimal reliability and are prone to subjective interpretations.

Procedural issues of letters of recommendation

The primary studies that explored biases in narrative LoRs at the graduate level found evidence of: (1) gender and race biases (Biernat & Eidelman, 2007 ; Morgan et al., 2013 ); (2) bias arising from tone of LoRs (Posselt, 2018 ); (3) bias arising from admissions committees’ members being (un)familiar with the LoR writer (Posselt, 2018 ); (4) bias in admissions committees’ evaluations against underrepresented minority groups once applicants’ names are visible (Morgan et al., 2013 ). Requiring admissions committees to elaborate on their evaluations of narrative LoRs reduces biases (Morgan et al., 2013 ).

Acceptability of letters of recommendation

Two primary studies explored the acceptability of LoRs. One study showed that LoRs are the second most valued selection method in admissions to doctoral programs in the US context, because they shed light on applicants’ personal characteristics (Boyette-Davis, 2018 ). However, another study in the European context did not find that LoRs are given weight by admissions committees when they decide to reject or admit a student to a master’s program (MasterMind Europe, 2017 ). In the latter study, more than a half (58.3%) of surveyed applicants reported that they had to provide an LoR within their application file.

Validity of interviews

Evidence on validity of interviews in STEM graduate programs is limited to two studies. One focused on traditional interviews and the other on the highly structured and formalized form of interviews: multiple mini-interviews (MMIs). Traditional interviews do not allow to distinguish between most and least productive graduate students (in terms of their time to degree and number of first-author papers; Hall et al., 2017 ). However, MMIs allow to predict planning-related problematic study behavior (oude Egbrink & Schuwirth, 2016 ).

Procedural issues of interviews

No study addressed the procedural issues of interviews specifically in graduate admissions.

Acceptability of interviews

A survey among European masters’ programs demonstrated that interviews are used in 22.6% of English-taught masters’ programs across Europe (MasterMind Europe, 2017 ). Although it is not a widely used selection method, it is valued and regarded as a good practice by admissions committees. In addition, members of admissions committees reported that a poor interview is a reason for rejection in less than 5% of all cases. No studies were conducted on how favorable interviews are perceived by applicants to graduate programs.

Cost-effectiveness of interviews

Interviews can be expensive both for applicants and graduate school (Woo et al., 2023 ). Applicants may be required to travel and/or to take time off from their work for an interview. In addition, they usually take time to prepare for it. On the side of graduate schools, interviewing takes substantial time investment of admissions committees both for preparation and for conducting the interviews.

Personal statements (motivation letters)

Validity of personal statements.

A meta-analysis on predictive validity of personal statements showed that they were weak predictors of grades and faculty ratings and when considered together with the UGPA and standardized admissions tests, they provided no incremental validity (Murphy et al., 2009 ).

Procedural issues of personal statements

Woo et al. ( 2023 ) bring attention to the fact that financial and social capitals are of great asset for richer students who seek help in writing personal statements. The same authors indicate that prior research has shown that men tend to use more acting and self-promotional tone in writing than females, which can have direct effects for creating biases in graduate admissions toward men (Woo et al., 2023 ).

Acceptability of personal statements

Personal statements are used frequently (MasterMind Europe, 2017 ) and are required from international applicants almost twice as often as from internal applicants (i.e., those, who obtained a bachelor’s degree at the same institution; MasterMind Europe, 2017 ). Personal statements are used to assess students’ motivation, make inferences about personal qualities, previous academic background, and cognitive ability (Kurysheva et al., 2019 ), provide information on whether a student’s background will contribute to the diversity of the student body (Posselt, 2014 ).

In most cases, personal statements did not serve as a reason for failure in the admissions process, according to members of admissions committees (MasterMind Europe, 2017 ).

Intelligence assessments

Validity of intelligence assessments.

Intelligence assessments are significantly correlated with academic performance (defined as grades, results of educational tests, and procedural and declarative knowledge; Poropat, 2009 ; Schneider & Preckel, 2017 ).

Procedural issues of intelligence assessments

Practical utility of intelligence as a predictor of study success is usually reduced, because it overlaps significantly with measures of prior performance (e.g., grades; Poropat, 2009 ).

Acceptability of intelligence assessments

In a cross-sectional study on the samples of students in the life sciences and natural sciences, it was shown that admissions criteria related to intelligence play a moderately important role in admissions decisions along with several other admissions criteria (Kurysheva et al., 2019 ). However, those admissions committees participating in the study did not apply specific intelligence assessments in their programs; the inferences on student intelligence were made from other selection methods rather than specific intelligence testing (Kurysheva et al., 2019 ).

Personality assessments

Validity of personality assessments.

The most common personality assessment is based on the five-factor model named the “Big Five”. It distinguishes five primary factors of personality (Goldberg, 1993 ): (1) conscientiousness, and it is one of the most stable findings both from individual and meta-analytical studies that conscientiousness is a medium-to-large predictor of study success (Butter & Born, 2012 ; Poropat, 2009 ; Schneider & Preckel, 2017 ; Trapmann et al., 2007 ; Walsh, 2020 ); (2) agreeableness, with mixed findings regarding its predictive value; (3) openness to experience, also has mixed findings, (4) neuroticism with no significant relation to study success, (5) extraversion with no significant relation to study success (Poropat, 2009 ; Trapmann et al., 2007 ).

Other personal traits, not explicitly included in the Big Five, were also examined: (1) grit (defined as determination to achieve long-term goals), which does not explain additional variance in study success beyond conscientiousness (Walsh, 2020 ); (2) emotional intelligence, which has a weak-to-moderate effect on study success (Schneider & Preckel, 2017 ); (3) need for cognition (defined as an inclination to value activities that include effortful cognition), which has a weak-to-moderate effect on study success (Schneider & Preckel, 2017 ); (4) conscientiousness related to time management, so-called ecological conscientiousness, which is valid beyond the conventional Big Five in predicting Ph.D. performance criteria such as research progress, meeting deadlines, and probability to obtain a Ph.D. degree on time (Butter & Born, 2012 ).

Procedural issues of personality assessments

Two procedural issues of personality assessments are referred to in the context of graduate admissions: applicant faking and their coachability (Kyllonen et al., 2005 ). They arise from the fact that personality assessments are typically based on self-reports.

Acceptability of personality assessments

While graduate admissions committees regard personality assessment important to consider in principle (Kyllonen et al., 2005 ), they do not report to use them extensively (Boyette-Davis, 2018 ; MasterMind Europe, 2017 ).

Language proficiency assessments

Validity of language proficiency.

The available evidence on validity of different language assessments toward different dimensions of study success is presented in Table 5 .

Procedural issues of language proficiency assessments

No studies were detected that examined procedural issues of language proficiency assessments, specifically for graduate admissions.

Acceptability of language proficiency assessments

Four relevant aspects are worth noting: (1) In the European context, English language assessments were required mostly from foreign applicants to masters’ programs, although internal applicants are sometimes expected to submit them as well (MasterMind Europe, 2017 ); (2) Perceived importance of language proficiency by faculty members depended on a discipline: In humanities, for example, the importance is higher than in science disciplines (Lee & Greene, 2007 ); (3) Admissions committees usually limit the usage of language proficiencies assessments by checking whether the institutional cutoff score was met. Faculty members often expressed dissatisfaction with the language proficiency of admitted students, because some of them think that the cutoffs reflect not adequate but only minimal required language proficiency (Ginther & Elder, 2014 ); (4) Test takers do not seem to perceive TOEFL scores as a good indication of one’s language abilities (Mathews, 2007 ).

Prior research experience

Validity of prior research experience.

Prior research experience has been shown predictive for research skills performance (Gilmore et al., 2015 ), master’s and doctoral degree completion (Cox et al., 2009 ; Kurysheva et al., 2022 a), GGPA (Kurysheva et al., 2022 a; Kurysheva et al., 2022 b), faculty ratings (Weiner, 2014 ), time to degree (Kurysheva et al., 2022 a), but not for introductory graduate biomedical course (Park et al., 2018 ), graduate student productivity (Hall et al., 2017 ), time to degree (Hall et al., 2017 ). A meta-analysis showed that research experience during undergraduate studies, defined as a dichotomy “present” or “absent”, is unrelated to graduate study success (Miller et al., 2021 ).

Procedural issues of prior research experience

No studies examined procedural issues of prior research experience specifically in graduate admissions. However, there are concerns raised regarding usage of undergraduate research experience as a selection criterion as it might undermine diversity (Miller et al., 2021 ) or dilute the education mission of graduate curriculum (Kurysheva et al., 2022 a).

Acceptability of prior research experience

It appears that prior research experience is a valued component in graduate admissions (Boyette-Davis, 2018 ; Chari & Potvin, 2019 ). However, the extent of its importance depends on whether it is applied to a master’s or a doctoral program level (Chari & Potvin, 2019 ). The extent of importance of prior research experience also depends on what aspects are available for review. For example, simply having a basic level of research experience is significantly more important than having publications or conference participation records (Boyette-Davis, 2018 ).

Various graduate selection methods

In this category, the selection methods were collected that did not fall in previously reviewed categories: undergraduate institution selectivity, type of prior degree (bachelor’s or master’s), type of prior higher education institution, a rubric based on or a composite score of different selection methods, rate of progress, duration of prior studies and other specific assessment instruments.

Validity of various graduate selection methods

Undergraduate institution selectivity appears to have a positive relation to performance during the first semester of graduate studies (Moneta-Koehler et al., 2017 ; Park et al., 2018 ). Having a prior graduate degree increases the chances of graduate study success (Willcockson et al., 2009 ). The last four sub tables of Additional file 1 : Table S.3 (S3.26–S3.29) provide details into the findings of single studies on validation of all selection methods, which fell in this category.

Procedural issues of various graduate selection methods

Due to the scarcity of validation studies of the selection methods in this category, the procedural issues remain underexamined. One study addressed academic pedigree as a procedural issue of undergraduate institution selectivity (Posselt, 2018 ). Academic pedigree is the belief that higher rank of prior HEI signifies stronger student performance potential. In case of academic pedigree, the grades might be interpreted within the context of how rigorous the student’s curriculum was at a prior HEI. However, it appears that the selectivity and reputation of prior HEI are not clearly stated but somewhat hidden selection methods (Posselt, 2018 ). Posselt ( 2018 ) underscored that “privileging elite academic pedigrees in graduate admissions preserves racial and socioeconomic inequities that many institutions say they wish to reduce” (p. 497).

Acceptability of various graduate selection methods

Acceptability of selection methods in this category varies. The decisive factors in admissions by graduate admissions committees are as follows: certain undergraduate courses, type of prior academic background, type of prior education institution (Chari & Potvin, 2019 ).

Other selection methods, even if required, were not given substantial weight in selection decisions (Boyette-Davis, 2018 ; MasterMind Europe, 2017 ). Among them are extracurricular activities, teaching experience, quantitative skills, work experience, curriculum vitae (CV), photographs, essays, time management skills, understanding social relevance of research, evidence of integrity. Applicants seem to accept well selection methods that consist of different scales, even if the scales concern questions ranging from scientific knowledge to motivation (van Os, 2007 ).

Cost-effectiveness of various graduate selection methods

Using a total score of a rubric that combines different selection methods substantially increases the admissions rate of underrepresented students without increasing the time investment of admissions committees (Young et al., 2023 ).

This study, which focuses on the available research between 2005 and 2023, is the first review on both cognitive and noncognitive selection methods in graduate education and focuses on STEM disciplines. Studies dedicated to reliability and cost-effectiveness of graduate selection methods were rarely conducted during the examined time span. Therefore, the review’s focus was on integrating research evidence on the three evaluative quality principles of predictive validity, acceptability, and procedural issues.

Summary: key findings

Figure  2 provides a visualization of the selection methods located according to the extent of their predictive validity and acceptability by admissions committees. The dimensions of acceptability by applicants or procedural issues are not depicted, because this would require a third and fourth dimensions which would make the figure more difficult to interpret.

Note . The location of selection criteria (in a larger font) and the respective dimensions of study success (in a smaller font) are approximations based on the findings of the review. The colors refer to the X -axis: Red is used for selection methods that are invalid toward respective dimensions of study success. Green is used for selection methods that are valid toward respective dimensions of study success

Summary of the findings on two evaluative quality principles: validity and acceptability by admissions committees

The key findings of this review relate to three main evaluative quality principles we examined. The first key finding is that the predictive validity of applied selection methods varies substantially. The medium-to-strong predictors of several graduate study success dimensions are (1) prior grades (including UGPA), (2) GRE General, (3) intelligence assessments, and (4) the personality trait conscientiousness. The following selection methods are also valid, but to a lesser extent: (1) letters of recommendation, (2) tests on language proficiency, (3) personality aspects such as emotional intelligence and need for cognition, (4) undergraduate research experience (when defined as a grade for undergraduate thesis, duration of research project, but not as dichotomous absence or presence of research experience), and (5) MMI (based on limited amount of research). The selection methods in graduate admissions with lack of predictive validity were also detected: (1) personal statements, (2) traditional interviews, and (3) two personal traits (extraversion and neuroticism). This review highlights that the specific selection methods (e.g., the GRE General and UGPA) would appear valid toward certain dimensions of study success (e.g., GGPA) but not the others (e.g., research productivity).

The second key finding shows that the main procedural issues of selection methods are admissions biases, faking, coaching effects, item position effects, test preparation, and stereotype threat. While for some of the methods, the procedural issues constitute a prominent research debate (e.g., a debate on biases involved in implementation of the GRE), the procedural issues of others have not been adequately addressed (e.g., imperfections of grade conversion).

The third finding is that some invalid selection methods are widely accepted by admissions committees, while a similar method with a more structured format and with preliminary indications for validity does not appear to be widespread in STEM admissions. For example, personal statements appear to have negligible validity, especially in the presence of other selection methods but are still widely used (see Fig.  2 ).

Some evidence from outside of STEM graduate admissions

It is important to note that there is profound research on procedural issues and acceptability of selection methods outside of graduate admissions, namely, in undergraduate admissions and personnel selection. They were not included in results, because they did not fulfill inclusion criteria for this review. However, they are worth mentioning here in the discussion section, because it is unlikely that the procedural issues of the same selection method such as biases, faking, or coaching would be heavily determined by the education level. The following two subsections (procedural issues and acceptability) will, therefore, be dedicated to the outline of those procedural issues and acceptability of some selection methods that received little attention in graduate admissions but were investigated in undergraduate admissions and personnel selection.

Procedural issues

Procedural issues of (traditional) interviews . The current review did not detect studies on procedural issues of interviews in graduate STEM admissions. However, the findings from undergraduate, graduate nonSTEM, and personnel selection research are as follows.

The first procedural issue is susceptibility of interviews to biases toward gender, disability status, and ethnicity. Biases during interviews might come into play at different moments starting from so-called rapport building (a “small chat” aimed at helping applicants to feel comfortable), through the interview itself, and during the evaluation stage after the interview has ended (Levashina et al., 2014 ). Reducing bias and increasing validity and reliability of interviews is possible through introducing structure and different formats of interview: for example, phone or video interviews are more adaptable for structuring than face-to-face interviews (Levashina et al., 2014 ).

The second procedural issue is susceptibility of interviews to subjective interpretations of student “soft variables”, such as motivation. A study on a sample of students in a selective college in the Netherlands demonstrated that scores on interviews contribute little to prediction of study success but create risk of subjective interpretations. For example, many of the students whom the interviewers indicated were at risk of expulsion finished their first year successfully (Reumer & van der Wende, 2010 ). The authors note that “interviews provide extra guidance to both the student and the institution as to whether the student is choosing the right study program (and not so much as whether he is able to complete it successfully)” (Reumer & van der Wende, 2010 , p. 20).

The third procedural issue of interviews is faking by applicants, defined as “the conscious distortions of answers to the interview questions to obtain a better score on the interview and/or otherwise create favorable perceptions” (Levashina & Campion, 2007 , p. 1639). Among undergraduate job applicants, the estimates of faking, understood in the above-defined broad sense, are as high as 90%, and the estimates of faking that is closer to lying range from 28 to 75% (Levashina & Campion, 2007 ).

The fourth procedural issue is impression management strategy used by some applicants (e.g., constant smiling), which contributes to admissions committees’ perception of these applicants as “glowing” and having “a very nice personality” (Posselt, 2016 , p. 144). The fifth procedural issue of interviews is that they provoke a broader actual evaluation of applicants than is formally communicated. For example, it has been shown that sometimes admissions committees’ distrust language skills of certain groups of international applicants, and therefore, they use the interview as an additional language check, while proclaiming that they want to assess applicants’ knowledge on the subject (Posselt, 2016 ).

The fifth procedural issue is susceptibility of interviews to weight bias. It was shown that applicants with higher body mass index (BMI) were admitted to a graduate psychology program less frequently than students with lower BMI, and this difference is especially prominent for female applicants (Burmeister et al., 2013 ).

Procedural issues of personal statements . In the literature outside of STEM graduate selection, namely, in the medical education programs, the biases of gender, age, socioeconomic class, country of origin, and ethnicity were shown to be present in admissions committees’ evaluations of personal statements (for the description, see the review of Kuncel et al., 2020 ).

Procedural issues of personality assessments . Similar to findings in graduate admissions, researchers who conducted studies in undergraduate and personnel selection show that the major procedural issue appears to be faking (Birkeland et al., 2006 ; König et al., 2017 ; Pavlov et al., 2019 ). The extent of faking depends on personality dimension under examination, type of test, aimed position (Birkeland et al., 2006 ), and situation stakes (Pavlov et al., 2019 ). However, there are approaches, where supervisors of students are asked to report on their personality, and while the supervisors also tend to fake when reporting on the personality of their students, the extent of their faking is smaller (König et al., 2017 ).

• Acceptability

In personnel selection, a review was conducted on how favorable different selection methods are rated by job applicants. From the review, it appears that the most preferred methods are work sample and interviews; overall favorably evaluated selection methods are resumes, cognitive tests, references, and personality assessments. The least preferred are honesty tests, personal contacts, and graphology (Anderson et al., 2010 ). Each selection method was assessed on several acceptability scales. For example, perceived scientific validity of LoRs is low, but their interpersonal warmth is high. In contrast to LoRs, intelligence assessments are perceived high on scientific validity and respectful of privacy but low on interpersonal warmth (Anderson et al., 2010 ). Interestingly, when it comes to structure of interviews, both applicants and interviewers perceive structured interviews less positively than unstructured interviews (Levashina et al., 2014 ). Similar to interviews, applicants perceive personality assessments favorably, especially the dimension “opportunity to perform” (Anderson et al., 2010 ).

Graduate selection methods as a distinct area for research

This review maps research evidence on selection methods used specifically at the graduate level. Several selection instruments that are used in admissions to professional schools such as medical school (e.g., situational judgment tests, MMIs, and selection centers) are not used in graduate STEM admissions. What are the potential reasons for this difference? The most obvious difference is that admissions to professional schools are directed toward detecting certain skills and traits of applicants to predict key competencies which are different from those of STEM researchers. The frameworks have been developed that define key competencies in medical profession (e.g., the Canadian Medical Education Directives for Specialists). They specify the knowledge, skills, abilities, and other characteristics (KSCAOs), related to competent performance within certain healthcare professions (for example, see Kerrin et al., 2018 ). Like medical education, graduate STEM education is also confronted with the question of which KSCAOs define an engineer or a researcher in STEM fields. A more general question would be even broader: whether a person is a researcher or a professional or not—and if not, why not? Does this have to do with academic freedom of researchers (Vrielink et al., 2011 ) and their roles as producers of critical knowledge, contributors to expansive learning, and organizers of a space for dialogue (Miettinen, 2004 )? Do the existing selection instruments reviewed in this study adequately capture prerequisites for competent performance on researchers’ roles? Are there any other selection methods that have potential to do this better? This review might, therefore, be regarded only as one of the first steps toward getting closer to answering such questions.

Implications for research and practice

Implications for research . This review has revealed significant gaps in the existing research, with an extremely low number of papers examining certain selection methods that appear to demonstrate medium and strong validity in graduate education. For example, the validity of MMIs, last-year GPA, and prior research experience have all been investigated in single studies, and the results are promising. To draw more meaningful conclusions, researchers in the field of student selection may wish to study the validity and other evaluative quality principles of these methods across a range of student populations and disciplines.

Implications for practice . From our review, it appeared that the selection methods that have no predictive value in graduate student selection are (1) personal statements; (2) traditional interviews; (3) narrative recommendation letters. Therefore, it is advised to avoid these instruments when making admissions decisions. This, however, does not mean that these instruments cannot be used for other purposes. For example, personal statements may be used for encouraging students to reflect on their motivation for a specific program and getting acquainted with it through exploration of the program’s curriculum, internship opportunities, and career perspectives (Wouters et al., 2014 ).

The variety of selection methods which practitioners should consider including in their selective admissions to research masters’ programs in STEM are as follows: (1) undergraduate grade point average (UGPA), (2) GRE General, (3) standardized language tests, such as TOEFL.

With additional caution, the following methods could be considered: (1) prior research experience (for admissions to research graduate programs); (2) GPA for the last year of a bachelor’s program; (3) standardized recommendation letters; (4) multiply mini-interviews; (5) standardized certified intelligence assessments; (6) assessments of (ecological) conscientiousness.

Inclusion of each of these selection methods should be guided by understanding which dimensions of study success these selection methods are capable of predicting, whether a selection method is accepted (and to what extent) by admissions committees and applicants, and whether the admissions committees are aware of the correct usage of a selection method.

Future directions

The methodological approach toward researching selective admissions.

In most of the primary studies reviewed, the regression approach was used. While it is a widely accepted type of analysis in this field, it is limited, because the findings on amount of explained variance are usually hard to interpret. Moreover, the findings based on the regression approach do not allow one to set the cutoff scores. Future research would benefit from applying other methodologies. For example, Bridgeman et al. ( 2009 ) offer a method that divides students within a department into quartiles based on a selection method of interest and a dimension of study success. The methodology that allows (under certain conditions) the establishment of cutoff scores for selective admissions methods is the Signal Detection Theory (van Ooijen-van der Linden, 2017 ). Finally, future research approaches toward selection methods should account for a multilevel and dynamic nature of student selection (Patterson et al., 2018 ) as well as the importance of other evaluative quality principles of selection methods not addressed in this review, such as practicality/administrative convenience, ease of interpretation, and so forth (see for the full list Patterson and Ferguson, 2010 ).

Future directions in practice of selective admissions

Research evidence on selection methods has advanced significantly in recent years. In some national and institutional contexts, the research findings are actively being translated into practice (e.g., Council of Graduate Schools, 2021 ). However, along with that, “today’s faculty choose students on the basis of an array of perceptions that only sometimes have a strong evidentiary basis” (Posselt, 2016 , p. 176). Therefore, professionalization of admissions staff and formation of communities of good admissions practices are required. Even despite certain gaps in research, already existing evidence allows significant progress toward the evidence-based policy on selective admissions for graduate schools across the world.

In addition to professionalization of admissions staff, it is important to consider monitoring and evaluation of the admissions process: Is there a closed-loop control of the admissions process? Are the selection methods scrutinized adequately in accreditation? Is there sufficient reporting on the chosen admissions process and selection methods applied in the HEI to higher levels? Ultimately, the answers to these questions reflect the extent of accountability of admissions committees for the soundness of their admissions practices. Accountability would imply reporting on data on each selection round to higher levels within HEI’s organization. Institutional research, in turn, could have a role in analyzing emerging patterns, testing these against relevant models, and giving warning signals when substantial deviations occur. This would contribute to an adaptive admissions process that could eventually lead to fairer and more objective graduate admissions (Zimmermann et al., 2017a , 2017b ).

Selective admissions and societal responsibility

Considering increasing numbers of applications and capacity limitations at research universities, evidence-based student selection is increasingly recognized as a socially significant practice which should diminish rather than enhance inequality. Failing to meet requirements of fairness, objectiveness, and transparency primarily leads to missed opportunities for capable students and a HEI, the inability of a HEI to justify the selection decisions, jeopardizing the diversity of the student body, infringement of students’ rights on equal access to higher education, and the loss of time and efforts both by students and institutions. In extreme cases, abandoning quality requirements toward selective admissions process might lead to appearances of criminal bribing schemes (e.g., the 2019 college admissions bribery scandal in the US). Designing a sound admissions process for graduate level education is, therefore, a necessary step for preventing these issues from arising or to cease their existence entirely. Finally, student selection has become an increasingly politicized societal topic, where advocacy groups and politicians are actively participating. In some countries, the alternatives to selective admissions are discussed, such as re-introducing the (weighted) lottery system in the Netherlands as a more neutral solution (The national government of the Netherlands, 2021 ). However, there is some critique of its effect on equal access, because a weighted lottery is based on selection criteria as well (Council of State of the Netherlands, 2021 ).

Limitations

Drawing conclusions from a large number of papers inevitably brings a risk of losing the nuances of each study (see Additional file 1 : Table S3 for more details). It also means that the samples of studies on predictive validity of graduate selection methods in several instances included not only STEM students but also students from other disciplines. Even if the strength of the relationship between a selection method and various dimensions of graduate study success is diluted by inclusion of students from other disciplines, it is unlikely that the direction of relationship would be the opposite. From this, however, an advantage appeared that the findings of this review to a certain extent are generalizable to other academic disciplines within graduate levels of education.

Another limitation is that our inferences on the effects sizes (negligible, small, medium, and strong effect sizes) were based on the interpretations of the studies’ authors. To refine the estimations of the effect sizes, the meta-analyses on reviewed selection methods would be required. Such goals were outside the scope of this review; however, the indications that this review provides are robust enough to answer the main question on whether a selection method is valid in principle.

Furthermore, most studies on the topic were carried out in the US, which has inevitably influenced this review. Therefore, practitioners and policymakers outside the US should account for this unintentional bias when referring to the results and conclusions of this review. However, we think that the cultural/geographical bias may have mainly impacted the results and conclusions related to acceptability of selection methods as it addresses individuals’ perceptions, which are more easily affected by culture. On the other hand, we think that (a) validity and (b) procedural issues of selection methods are much less affected by cultural/geographical bias, because these evaluative quality principles relate to (a) the predictive power toward uniformed dimensions of study success and (b) concerns involved in using certain selection methods. For example, a common concern regarding richer applicants having more financial possibilities than poorer applicants to be coached on standardized testing is relevant in any country.

Finally, the reviewed literature on acceptability of selection methods often contained evidence from admissions committees’ self-reports. Their reports could have been (un)consciously biased to a certain extent if they did not want to report, for example, the usage of invalid yet favored selection methods. Therefore, the observational ethnographic studies, like the one of Posselt ( 2016 ), gain special importance in this area of research: The observation might be a more appropriate method to detect “hidden” selection criteria and group dynamics within an admissions committee, because these concealed processes are influential toward admissions decisions.

The main aim of this review was to collect, map, synthesize, and critically analyze the available research evidence on graduate selection methods with a focus on STEM disciplines. The results of the systematic search of research literature were categorized according to a type of selection method and core evaluative quality principles (predictive validity, acceptability, and procedural issues). Ten categories of graduate selection methods emerged. It was found that the predictive validity of prior grades, GRE General, intelligence assessments, and conscientiousness toward several study success dimensions is of medium-to-strong extent. Letters of recommendation, tests on language proficiency, emotional intelligence, and need for cognition are valid as well, but of weak-to-medium extent. Based on the limited evidence, it also appears that prior research experience, multiple mini-interviews, and selectivity of prior institution might have significant relationships with certain dimensions of graduate study success. Personal statements, traditional interviews, and personal traits such as extroversion and neuroticism are invalid predictors of graduate study success.

When choosing the selection methods to be applied in the admissions process, policy makers and admissions committees should use only valid instruments. They should also be aware of typical applicant reactions toward these methods as well as procedural issues, such as possible adverse effects toward certain groups, susceptibility for biases, faking, coaching, and stereotype threat. The admissions committees are advised (1) to completely exclude invalid selection instruments from their admissions requirements, (2) to define the dimensions of study success that are most important for their program, (3) to use those selection methods that showed predictive validity toward these predefined study success dimensions, accounting for applicant reactions and procedural issues of each of those methods, and (4) to ensure the accountability of the admissions process by reporting on data on each selection round to higher levels within HEI’s organization, which should in turn conduct further analysis and regular evaluations of admissions processes.

Availability of data and materials

The set of articles used during the current review are available from the corresponding author on reasonable request.

Abbreviations

The Computerized Enhanced ESL Placement Test

Curriculum Vitae

Education Resources Information Center

Examen Nacional de Ingreso al Posgrado

Grade Point Average

Graduate Record Examinations General Test

The Quantitative Reasoning measure of the GRE General Test

The Verbal Reasoning measure of the GRE General Test

The Analytical Writing measure of the GRE General Test

Higher Education Institution

International English Language Testing System

Organization for Economic Co-operation and Development

A Doctor of Philosophy

Letters of Recommendation

Multiple Mini-Interview

Science, Technology, Engineering, and Math

Test of English as a Foreign Language

Undergraduate Grade Point Average

Studies that met inclusion criteria and were included in the “Results” section of this review are marked with an asterisk.

*Álvarez-Montero, F., Mojardin-Heraldez, A., & Audelo-Lopez, C. (2014). Criteria and instruments for doctoral program admissions. Electronic Journal of Research in Educational Psychology, 12 (3), 853–866. https://www.researchgate.net/publication/266200011_Criteria_and_instruments_for_doctoral_program_admission

Anderson, N., Salgado, J. F., & Hülsheger, U. R. (2010). Applicant reactions in selection: Comprehensive meta-analysis into reaction generalization versus situational specificity. International Journal of Selection and Assessment, 18 (3), 291–304. https://doi.org/10.1111/j.1468-2389.2010.00512.x

Article   Google Scholar  

*Attali, Y., & Sinharay, S. (2015). Automated trait scores for GRE ® writing tasks: Automated trait scores for GRE ® writing tasks. (Report No. RR-15-15). Educational Testing Service . https://doi.org/10.1002/ets2.12062

*Biernat, M., & Eidelman, S. (2007). Translating subjective language in letters of recommendation: The case of the sexist professor. European Journal of Social Psychology, 37 (6), 1149–1175. https://doi.org/10.1002/ejsp.432

Birkeland, S. A., Manson, T. M., Kisamore, J. L., Brannick, M. T., & Smith, M. A. (2006). A meta-analytic investigation of job applicant faking on personality measures: Job applicant faking on personality measures. International Journal of Selection and Assessment, 14 (4), 317–335. https://doi.org/10.1111/j.1468-2389.2006.00354.x

*Bleske-Rechek, A., & Browne, K. (2014). Trends in GRE scores and graduate enrollments by gender and ethnicity. Intelligence, 46 , 25–34. https://doi.org/10.1016/j.intell.2014.05.005

*Boyette-Davis, J. (2018). A data-based assessment of research-doctorate programs in the United States. The Journal of Undergraduate Neuroscience Education, 17 (1), A54–A58. https://doi.org/10.17226/12994

*Bridgeman, B., Burton, N., & Cline, F. (2009). A note on presenting what predictive validity numbers mean. Applied Measurement in Education, 22 (2), 109–119. https://doi.org/10.1080/08957340902754577

*Briihl, D. S., & Wasieleski, D. T. (2007). The GRE analytical writing test: Description and utilization. Teaching of Psychology, 34 (3), 191–193. https://doi.org/10.1080/00986280701498632

Burmeister, J. M., Kiefner, A. E., Carels, R. A., & Musher-Eizenman, D. R. (2013). Weight bias in graduate school admissions. Obesity, 21 (5), 918–920. https://doi.org/10.1002/oby.20171

*Burmeister, J., McSpadden, E., Rakowski, J., Nalichowski, A., Yudelev, M., & Snyder, M. (2014). Correlation of admissions statistics to graduate student success in medical physics. Journal of Applied Clinical Medical Physics, 15 (1), 375–385. https://doi.org/10.1120/jacmp.v15i1.4451

Burton, N. W., & Wang, M. (2005). Predicting long-term success in graduate school: A collaborative validity study. (Report No. 99-14R. ETS RR-05-03). Educational Testing Service. http://grad.uga.edu/wpcontent/uploads/2017/09/GRE_Research_Report.pdf

*Butter, R., & Born, MPh. (2012). Enhancing criterion-related validity through bottom-up contextualization of personality inventories: The construction of an ecological conscientiousness scale for PhD candidates. Human Performance, 25 (4), 303–317. https://doi.org/10.1080/08959285.2012.703730

*Camara, W., Packman S., & Wiley A. (2013). College, graduate, and professional school admissions testing. In K. F. Geisinger, B. A. Bracken, J. F. Carlson, J.-I. C. Hansen, N. R., Kuncel, S. P., Reise, & M. C. Rodriguez (Eds.), APA handbook of testing and assessment in psychology: Testing and assessment in school psychology and education (Vol 3, pp. 297–318). American Psychological Association. https://doi.org/10.1037/14049-014

*Chari, D., & Potvin, G. (2019). Admissions practices in terminal master’s degree-granting physics departments: A comparative analysis. Physical Review Physics Education Research, 15 (1), Article 010104. https://doi.org/10.1103/PhysRevPhysEducRes.15.010104

*Cho, Y., & Bridgeman, B. (2012). Relationship of TOEFL iBT® scores to academic performance: Some evidence from American universities. Language Testing, 29 (3), 421–442. https://doi.org/10.1177/0265532211430368

*Cline, F., & Powers, D. (2014). Test-taker perceptions of the role of the GRE® General test in graduate admissions: Preliminary findings. In The Research Foundation for the GRE revised general test: A compendium of studies (p. 6.1.1–6.1.6). Educational Testing Service. https://www.ets.org/s/research/pdf/gre_compendium.pdf

Council of Graduate Schools. (2021). CGS best practices programs in graduate admissions and enrollment management . https://cgsnet.org/admissions-and-recruitment

Council of State of the Netherlands [Raad van State]. (2021). Amendment of the Higher Education and Scientific Research Act in relationship to the addition of decentralized draw as a selection method for higher education programs with fixed capacity [Wijziging van de wet op het hoger onderwijs en wetenschappelijk onderzoek in verband met het toevoegen van decentrale loting als selectiemethode voor opleidingen met capaciteitsfixus in het hoger onderwijs]. (W05.20.0508/I). https://www.raadvanstate.nl/@123920/w05-20-0508/

*Cox, G. W., Hughes, W. E., Jr., Etzkorn, L. H., & Weisskopf, M. E. (2009). Predicting computer science PhD completion: A case study. IEEE Transactions on Education, 52 (1), 137–143. https://doi.org/10.1109/TE.2008.921458

*Davey, T., & Lee, Y.-H. (2011). Potential impact of context effects on the scoring and equating of the multistage GRE revised General test . (Report No. GREB-08–01). Educational Testing Service. https://doi.org/10.1002/j.2333-8504.2011.tb02262.x

De Boer, T., & Van Rijnsoever, F. (2022a). In search of valid non-cognitive student selection criteria. Assessment & Evaluation in Higher Education, 47 (5), 783–800. https://doi.org/10.1080/02602938.2021.1958142

*De Boer, T., & Van Rijnsoever, F. J. (2022b). One field too far? Higher cognitive relatedness between bachelor and master leads to better predictive validity of bachelor grades during admission. Assessment & Evaluation in Higher Education. https://doi.org/10.1080/02602938.2022.2158453

DeClou, L. (2016). Who stays and for how long: examining attrition in Canadian graduate programs. Canadian Journal of Higher Education, 46 (4), 174–198.

De Wit, H., & Altbach, P. G. (2020). Internationalization in higher education: global trends and recommendations for its future. Policy Reviews in Higher Education, 5 (1), 28–46. https://doi.org/10.1080/23322969.2020.1820898

European Grade Conversion System. (2020). Grade conversion–an introduction . http://egracons.eu/page/about-egracons-project-and-tool

*Fischer, F. T., Schult, J., & Hell, B. (2013). Sex-specific differential prediction of college admissions tests: A meta-analysis. Journal of Educational Psychology, 105 (2), 478–488. https://doi.org/10.1037/a0031956

Garaz, S., & Torotcoi, S. (2017). Increasing access to higher education and the reproduction of social inequalities: The case of Roma university students in Eastern and Southeastern Europe. European Education, 49 (1), 10–35. https://doi.org/10.1080/10564934.2017.1280334

*Garces, L. M. (2014). Aligning diversity, quality, and equity: The implications of legal and public policy developments for promoting racial diversity in graduate studies. American Journal of Education, 120 (4), 457–480. https://doi.org/10.1086/676909

*Gilmore, J., Vieyra, M., Timmerman, B., Feldon, D., & Maher, M. (2015). The relationship between undergraduate research participation and subsequent research performance of early career STEM graduate students. The Journal of Higher Education, 86 (6), 834–863. https://doi.org/10.1353/jhe.2015.0031

*Ginther, A., & Elder, C. (2014). A comparative investigation into understandings and uses of the TOEFL iBT® test, the international English language testing service (academic) test, and the Pearson test of English for graduate admissions in the United States and Australia: A case study: An investigation into test score understandings and uses. (Report No. RR– 14-44). Educational Testing Service . https://doi.org/10.1002/ets2.12037

Goldberg, L. R. (1993). The structure of phenotypic personality traits. American Psychologist, 48 (1), 26–34. https://doi.org/10.1037/0003-066X.48.1.26

*Hall, J. D., O’Connell, A. B., & Cook, J. G. (2017). Predictors of student productivity in biomedical graduate school applications. PLoS ONE, 12 (1), e0169121. https://doi.org/10.1371/journal.pone.0169121

*Hausknecht, J. P., Halpert, J. A., Paolo, N. T. D., & Gerrard, M. O. M. (2007). Retesting in selection: A meta-analysis of practice effects for tests of cognitive ability. Journal of Applied Psychology, 92 , 373–385.

Howell, L. L., Sorenson, C. D., & Jones, M. R. (2014). Are undergraduate GPA and general GRE percentiles valid predictors of student performance in an engineering graduate program? International Journal of Engineering Education, 30( 5), 1145–1165. https://scholarsarchive.byu.edu/cgi/viewcontent.cgi?article=2342&context=facpub

Jayakumar, U. M., & Page, S. E. (2021). Cultural capital and opportunities for exceptionalism: Bias in university admissions. The Journal of Higher Education, 92 (7), 1109–1139. https://doi.org/10.1080/00221546.2021.1912554

Kerrin, M., Mossop, L., Morley, E., Fleming, G., & Flaxman, C. (2018). Role analysis: The foundation for selection systems. In F. Patterson & L. Zibarras (Eds.), Selection and recruitment in the healthcare professions: Research, theory and practice (pp. 139–165). Palgrave Macmillan.

Google Scholar  

Kirby, W., & van der Wende, M. (2019). The New Silk Road: Implications for higher education in China and the West? Cambridge Journal of Regions, Economy and Society, 12 (1), 127–144. https://doi.org/10.1093/cjres/rsy034

*Klieger, D. M., Cline, F. A., Holtzman, S. L., Minsky, J. L., & Lorenz, F. (2014). New perspectives on the validity of the GRE ® General test for predicting graduate school grades: New perspectives for predicting graduate school grades (Report No. RR– 14-26). Educational Testing Service . https://doi.org/10.1002/ets2.12026

König, C. J., Steiner Thommen, L. A., Wittwer, A.-M., & Kleinmann, M. (2017). Are observer ratings of applicants’ personality also faked? Yes, but less than self-reports. International Journal of Selection and Assessment, 25 (2), 183–192. https://doi.org/10.1111/ijsa.12171

*Kuncel, N. R., & Hezlett, S. A. (2007a). The utility of standardized tests: Response. Science, 316 , 1696–1697. https://doi.org/10.1126/science.316.5832.1694b

*Kuncel, N. R., & Hezlett, S. A. (2007b). Standardized tests predict graduate students’ success. Science, 315 (5815), 1080–1081. https://doi.org/10.1126/science.1136618

*Kuncel, N. R., & Hezlett, S. A. (2010). Fact and fiction in cognitive ability testing for admissions and hiring decisions. Current Directions in Psychological Science, 19 (6), 339–345. https://doi.org/10.1177/0963721410389459

Kuncel, N. R., Hezlett, S. A., & Ones, D. S. (2004). Academic performance, career potential, creativity, and job performance: Can one construct predict them all? Journal of Personality and Social Psychology, 86 (1), 148–161. https://doi.org/10.1037/0022-3514.86.1.148

*Kuncel, N. R., Kochevar, R. J., & Ones, D. S. (2014). A meta-analysis of letters of recommendation in college and graduate admissions: Reasons for hope. International Journal of Selection and Assessment, 22 (1), 101–107. https://doi.org/10.1111/ijsa.12060

*Kuncel, N. R., Wee, S., Serafin, L., & Hezlett, S. A. (2010). The validity of the Graduate Record Examination for master’s and doctoral programs: A meta-analytic investigation. Educational and Psychological Measurement, 70 (2), 340–352. https://doi.org/10.1177/0013164409344508

Kuncel, N., Tran, K., & Zhang, S. H. (2020). Measuring student character: Modernizing predictors of academic success. In M. E. Oliveri & C. Wendler (Eds.), Higher education admissions practices: An international perspective (pp. 276–302). Cambridge University Press.

Chapter   Google Scholar  

*Kurysheva, A., Koning, N., Fox, C. M., van Rijen, H. V., & Dilaver, G. (2022). Once the best student always the best student? Predicting graduate study success, using undergraduate academic indicators. Evidence from research masters’ programs in the Netherlands. International Journal of Selection and Assessment, 30 (4), 1–17. https://doi.org/10.1111/ijsa.12397

*Kurysheva, A., van Ooijen-van der Linden, L., van der Smagt, M. J., & Dilaver, G. (2022). The added value of signal detection theory as a method in evidence-informed decision-making in higher education: A demonstration. Frontiers in Education, 7 , Article 906611. https://doi.org/10.3389/feduc.2022.906611

*Kurysheva, A., van Rijen, H. V., & Dilaver, G. (2019). How do admission committees select? Do applicants know how they select? Selection criteria and transparency at a Dutch University. Tertiary Education and Management, 25 , 367–388. https://doi.org/10.1007/s11233-019-09050-z

*Kyllonen, P., Walters, A. M., & Kaufman, J. C. (2005). Noncognitive constructs and their assessment in graduate education: A review. Educational Assessment, 10 (3), 153–184. https://doi.org/10.1207/s15326977ea1003_2

Kyllonen, P. C., Walters, A. M., & Kaufman, J. C. (2011). The role of noncognitive constructs and other background variables in graduate education. (Report No. GREB-00-11). Educational Testing Service . https://doi.org/10.1002/j.2333-8504.2011.tb02248.x

*Lee, Y.-J., & Greene, J. (2007). The predictive validity of an ESL placement test: A mixed methods approach. Journal of Mixed Methods Research, 1 (4), 366–389. https://doi.org/10.1177/1558689807306148

Levashina, J., & Campion, M. A. (2007). Measuring faking in the employment interview: Development and validation of an interview faking behavior scale. Journal of Applied Psychology, 92 (6), 1638–1656. https://doi.org/10.1037/0021-9010.92.6.1638

Levashina, J., Hartwell, C. J., Morgeson, F. P., & Campion, M. A. (2014). The structured employment interview: Narrative and quantitative review of the research literature. Personnel Psychology, 67 (1), 241–293. https://doi.org/10.1111/peps.12052

*Lorden, J. F., Ed, Kuh, C. V., Ed, & Voytuk, J. A., Ed. (2011). Research-doctorate programs in the biomedical sciences: Selected findings from the NRC assessment. The National Academies Collection: Reports funded by National Institutes of Health. https://doi.org/10.17226/13213

*Lott, J. L. I., Gardner, S., & Powers, D. A. (2009). Doctoral student attrition in the STEM fields: An exploratory event history analysis. Journal of College Student Retention: Research, Theory and Practice, 11 (2), 247–266. https://doi.org/10.2190/CS.11.2.e

*MasterMind Europe. (2017). Admissions to English-taught programs (ETPs) at master’s level in Europe–Procedures, regulations, success rates and challenges for diverse applicants . ACA, StudyPortals, and Vrije Universiteit Amsterdam. http://mastermindeurope.eu/wp-content/uploads/2017/01/Report-2-Admissions-to-ETPs.pdf

*Mathews, J. (2007). Predicting international students’ academic success… may not always be enough: Assessing Turkey’s foreign study scholarship program. Higher Education, 53 (5), 645–673. https://doi.org/10.1007/s10734-005-2290-x

*Megginson, L. (2009). Noncognitive constructs in graduate admissions: An integrative review of available instruments. Nurse Educator, 34 (6), 254–261. https://doi.org/10.1097/NNE.0b013e3181bc7465

*Mendoza-Sanchez, I., deGruyter, J. N., Savage, N. T., & Polymenis, M. (2022). Undergraduate GPA predicts biochemistry PhD completion and is associated with time to degree. CBE—Life Sciences Education, 21 (2), ar19. https://doi.org/10.1187/cbe.21-07-0189

Merriam-Webster dictionary. (n.d.). Grade inflation. In Merriam-Webster.com dictionary . Retrieved February 18, 2022, from https://www.merriam-webster.com/dictionary/grade%20inflation

Miettinen, R. (2004). The roles of the researcher in developmentally-oriented research. In T. Kontinen (Ed.), Development intervention. Actor and activity perspectives (pp. 105–121). University of Helsinki, Center for Activity Theory and Developmental Work Research and Institute for Development Studies.

*Miller, E. M. (2019). Promoting student success in statistics courses by tapping diverse cognitive abilities. Teaching of Psychology, 46 (2), 140–145. https://doi.org/10.1177/0098628319834198

*Miller, A., Crede, M., & Sotola, L. K. (2021). Should research experience be used for selection into graduate school: A discussion and meta-analytic synthesis of the available evidence. International Journal of Selection and Assessment, 29 (1), 19–28. https://doi.org/10.1111/ijsa.12312

*Moneta-Koehler, L., Brown, A. M., Petrie, K. A., Evans, B. J., & Chalkley, R. (2017). The Limitations of the GRE in predicting success in biomedical graduate school. PLoS ONE, 12 (1), Article e0166742. https://doi.org/10.1371/journal.pone.0166742

*Morgan, W. B., Elder, K. B., & King, E. B. (2013). The emergence and reduction of bias in letters of recommendation: Bias in letters of recommendation. Journal of Applied Social Psychology, 43 (11), 2297–2306. https://doi.org/10.1111/jasp.12179

*Mupinga, E. E., & Mupinga, D. M. (2005). Perceptions of international students toward GRE. College Student Journal, 39 (2), 402–409.

*Murphy, K. R. (2009). How a broader definition of the criterion domain changes our thinking about adverse impact. In J. L. Outtz (Ed.), Adverse impact: Implications for organizational staffing and high stakes selection (pp. 137–160). Routledge. https://doi.org/10.4324/9780203848418

Murphy, S. C., Klieger, D. M., Borneman, M. J., & Kuncel, N. R. (2009). The predictive power of personal statements in admissions: A meta-analysis and cautionary tale. College & University, 84 (4), 83–88.

Okahana, H., & Zhou, E. (2018). International graduate applications and enrollment: Fall 2017 (pp. 1–24). Washington, DC: Council of Graduate Schools. https://cgsnet.org/Data-Insights/

*oude Egbrink, M. G. A., & Schuwirth, L. W. T. (2016). Narrative information obtained during student selection predicts problematic study behavior. Medical Teacher, 38 (8), 844–849. https://doi.org/10.3109/0142159X.2015.1132410

*Park, H.-Y., Berkowitz, O., Symes, K., & Dasgupta, S. (2018). The art and science of selecting graduate students in the biomedical sciences: Performance in doctoral study of the foundational sciences. PLoS ONE, 13 (4), Article e0193901. https://doi.org/10.1371/journal.pone.0193901

Patterson, F., & Ferguson, E. (2010). Selection for medical education and training. In T. Swanwick (Ed.), Understanding medical education (pp. 352–365). Wiley-Blackwell. https://doi.org/10.1002/9781444320282.ch24

Patterson, F., Knight, A., Dowell, J., Nicholson, S., Cousans, F., & Cleland, J. (2016). How effective are selection methods in medical education? A Systematic Review. Medical Education, 50 (1), 36–60. https://doi.org/10.1111/medu.12817

Patterson, F., Roberts, C., Hanson, M. D., Hampe, W., Eva, K., Ponnamperuma, G., Magzoub, M., Tekian, A., & Cleland, J. (2018). 2018 Ottawa consensus statement: Selection and recruitment to the healthcare professions. Medical Teacher, 40 (11), 1–9. https://doi.org/10.1080/0142159X.2018.1498589

Pavlov, G., Maydeu-Olivares, A., & Fairchild, A. J. (2019). Effects of applicant faking on forced-choice and Likert scores. Organizational Research Methods, 22 (3), 710–739. https://doi.org/10.1177/1094428117753683

*Payne, D. (2015). A common yardstick for graduate education in Europe. Journal of the European Higher Education Area, 2 , 21–48.

*Poropat, A. E. (2009). A meta-analysis of the five-factor model of personality and academic performance. Psychological Bulletin, 135 (2), 322–338. https://doi.org/10.1037/a0014996

*Posselt, J. R. (2014). Toward inclusive excellence in graduate education: Constructing merit and diversity in PhD admissions. American Journal of Education, 120 (4), 481–514. https://doi.org/10.1086/676910

Posselt, J. R. (2016). Inside graduate admissions: Merit, diversity, and faculty gatekeeping . Harvard University Press.

Book   Google Scholar  

*Posselt, J. R. (2018). Trust Networks: A new perspective on pedigree and the ambiguities of admissions. The Review of Higher Education, 41 (4), 497–521. https://doi.org/10.1353/rhe.2018.0023

*Powers, D. E. (2005). Effects of preexamination disclosure of essay prompts for the GRE analytical writing assessment. (Report No. 01-07R). Educational Testing Service . https://doi.org/10.1002/j.2333-8504.2005.tb01978.x

*Powers, D. E. (2017). Understanding the impact of special preparation for admissions tests. In R. E. Bennett & M. von Davier (Eds.), Advancing human assessment. The methodological, psychological and policy contributions of ETS (pp. 553–564). Springer Open. https://doi.org/10.1007/978-3-319-58689-2

Proudfoot, S., & Hoffer, T. B. (2016). Science and engineering labor force in the US. In L. Gokhberg, N. Shmatko, & L. Auriol (Eds.), The science and technology labor force (pp. 77–120). Springer International Publishing. https://doi.org/10.1007/978-3-319-27210-8

Reumer, C., & van der Wende, M. (2010). Excellence and diversity: The emergence of selective admissions policies in Dutch higher education. A case study on Amsterdam University College. Research & Occasional Paper Series: CSHE.15.10 , 1–28. https://cshe.berkeley.edu/publications/excellence-and-diversity-emergence-selective-admission-policies-dutch-higher-0

*Rock, J. L., & Adler, R. M. (2014). A descriptive study of universities’ use of GRE ® General test scores in awarding fellowships to first-year doctoral students: A descriptive study of universities’ use of GRE ® scores. (Report No. RR– 14–27). Educational Testing Service. https://doi.org/10.1002/ets2.12027

Salmi, J., & Bassett, R. M. (2014). The equity imperative in tertiary education: Promoting fairness and efficiency. International Review of Education, 60 (3), 361–377. https://doi.org/10.1007/s11159-013-9391-z

*Schneider, M., & Preckel, F. (2017). Variables associated with achievement in higher education: A systematic review of meta-analyses. Psychological Bulletin, 143 (6), 565–600. https://doi.org/10.1037/bul0000098

*Sealy, L., Saunders, C., Blume, J., & Chalkley, R. (2019). The GRE over the entire range of scores lacks predictive ability for PhD outcomes in the biomedical sciences. PloS One, 14 (3), e0201634. https://doi.org/10.1371/journal.pone.0201634

Sedlacek, W. E. (2003). Alternative admissions and scholarship selection measures in higher education. Measurement and Evaluation in Counseling and Development, 35 (4), 263–272. https://doi.org/10.1080/07481756.2003.12069072

The Bologna Declaration. Joint declaration of the European Ministers of Education, June 19, 1999. http://www.ehea.info/page-ministerial-conference-bologna-1999

The national government of the Netherlands [Rijksoverheid]. (2021). Loten voor studie zorgt voor kansengelijkheid [ Lots for study ensures equality of opportunity ]. https://www.rijksoverheid.nl/actueel/nieuws/2021/03/12/loten-voor-studie-zorgt-voor-kansengelijkheid

*Trapmann, S., Hell, B., Hirn, J.-O.W., & Schuler, H. (2007). Meta-analysis of the relationship between the Big Five and academic success at university. Zeitschrift Für Psychologie, 215 (2), 132–151. https://doi.org/10.1027/0044-3409.215.2.132

van Ooijen-van der Lindenvan der Smagt, L. M. J., Woertman, L., & te Pas, S. F. (2017). Signal detection theory as a tool for successful student selection. Assessment & Evaluation in Higher Education, 42 (8), 1193–1207. https://doi.org/10.1080/02602938.2016.1241860

*van Os, W. (2007). Selection to the master’s phase at the binary divide, a Dutch case study. Tertiary Education and Management, 13 (2), 127–140. https://doi.org/10.1080/13583880701238365

*Verostek, M., Miller, C. W., & Zwickl, B. (2021). Analyzing admissions metrics as predictors of graduate GPA and whether graduate GPA mediates Ph. D. completion. Physical Review Physics Education Research, 17 (2), 020115. https://doi.org/10.1103/PhysRevPhysEducRes.17.020115

Vrielink, J., Lemmens, P., & Parmentier, S. (2011). Academic freedom as a fundamental right. Procedia-Social and Behavioral Sciences, 13 , 117–141. https://doi.org/10.1016/j.sbspro.2011.03.009

*Walsh, M. J. (2020). Online doctoral student grade point average, conscientiousness, and grit: A moderation analysis. Journal of Educators Online , 17 (1). Advance online publication. https://www.thejeo.com/

Weedon, E. (2017). The construction of under-representation in UK and Swedish higher education: Implications for disabled students. Education, Citizenship and Social Justice, 12 (1), 75–88. https://doi.org/10.1177/1746197916683470

*Weiner, O. D. (2014). How should we be selecting our graduate students? Molecular Biology of the Cell, 25 (4), 429–430. https://doi.org/10.1091/mbc.e13-11-0646

*Weissman, M. B. (2020). Do GRE scores help predict getting a physics Ph.D.? A comment on a paper by Miller et al. Science Advances, 6 (23), Article eaax3787. https://doi.org/10.1126/sciadv.aax3787

*Westrick, P. A. (2017). Reliability estimates for undergraduate grade point average. Educational Assessment, 22 (4), 231–252. https://doi.org/10.1080/10627197.2017.1381554

*Willcockson, I. U., Johnson, C. W., Hersh, W., & Bernstam, E. V. (2009). Predictors of student success in graduate biomedical informatics training: Introductory course and program success. Journal of the American Medical Informatics Association, 16 (6), 837–846. https://doi.org/10.1197/jamia.M2895

*Wilson, M. A., DePass, A. L., & Bean, A. J. (2018). Institutional interventions that remove barriers to recruit and retain diverse biomedical PhD students. CBE—Life Sciences Education, 17 (2), Article 17:ar27. https://doi.org/10.1187/cbe.17-09-0210

*Wilson, M. A., Odem, M. A., Walters, T., DePass, A. L., & Bean, A. J. (2019). A model for holistic review in graduate admissions that decouples the GRE from race, ethnicity, and gender. CBE—Life Sciences Education, 18 (1), Article 18: ae7. https://doi.org/10.1187/cbe.18-06-0103

*Wollast, R., Boudrenghien, G., Van der Linden, N., Galand, B., Roland, N., Devos, C., De Clercq, M., Klein, O., Azzi, A., & Frenay, M. (2018). Who are the doctoral students who drop out? Factors associated with the rate of doctoral degree completion in universities. International Journal of Higher Education, 7 (4), 143–156. https://doi.org/10.5430/ijhe.v7n4p143

*Woo, S. E., LeBreton, J. M., Keith, M. G., & Tay, L. (2023). Bias, fairness, and validity in graduate-school admissions: A psychometric perspective. Perspectives on Psychological Science, 18 (1), 3–31. https://doi.org/10.1177/17456916211055374

Wouters, A., Bakker, A.H., van Wijk, I.J. et al. (2014). A qualitative analysis of statements on motivation of applicants for medical school. BMC Medical Education, 14 , 200. https://doi.org/10.1186/1472-6920-14-200

*Young, N. T., Tollefson, K., & Caballero, M. D. (2023). Making graduate admissions in physics more equitable. Physics Today, 76 (7), 40–45. https://doi.org/10.1063/PT.3.5271

Zimdars, A. M. (2016). Meritocracy and the university: Selective admission in England and the United States . Bloomsbury Publishing.

*Zimmermann, J., Heinimann, H. R., Brodersen, K. H., & Buhmann, J. M. (2015). A model-based approach to predicting graduate-level performance using indicators of undergraduate-level performance. Journal of Educational Data Mining, 7 (3), 151–176. https://doi.org/10.5281/zenodo.3554733

*Zimmermann, J., von Davier, A. A., Buhmann, J. M., & Heinimann, H. R. (2017a). Validity of GRE General test scores and TOEFL scores for graduate admissions to a technical university in Western Europe. European Journal of Engineering Education, 43 (1), 144–165. https://doi.org/10.1080/03043797.2017.1343277

Zimmermann, J., von Davier, A., & Heinimann, H. R. (2017b). Adaptive admissions process for effective and fair graduate admissions. International Journal of Educational Management, 31 (4), 540–558. https://doi.org/10.1108/IJEM-06-2015-0080

Download references

Acknowledgements

The authors would like to thank Prof. Dr. Marijk van der Wende for her supervision of the overarching PhD project focused on graduate selective admissions, where this study was part of. The authors are grateful to Adam Frick and Dr. Christine Merie Fox for copy editing.

This study received internal institutional funding.

Author information

Authors and affiliations.

Center of Education and Training, University Medical Center Utrecht, HB-4.05, P.O. Box 85500, 3508 GA, Utrecht, Netherlands

Anastasia Kurysheva, Harold V. M. van Rijen, Cecily Stolte & Gönül Dilaver

Graduate School of Life Sciences, Biomedical Sciences department, Utrecht University, HB-4.05, P.O. Box 85500, 3508 GA, Utrecht, Netherlands

You can also search for this author in PubMed   Google Scholar

Contributions

AK, HvR, and GD developed the original concept and defined the inclusion criteria. AK outlined, conducted the systematic search of literature, collected and coded the articles, wrote the original draft, and conducted a revision. AK and SC worked together on the coding framework. SC acted as the second coder and helped with the calculation of inter-rater agreement. All authors worked together to revise the article prior to submission. All authors read and approved the final manuscript.

Corresponding author

Correspondence to Anastasia Kurysheva .

Ethics declarations

Competing interests.

The authors declare that they have no competing interests.

Additional information

Publisher's note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Additional file 1: table s1..

Key words used in the search in literature data bases. Table S2. Number of articles relating to each selection method and evaluative quality principle under consideration. Table S3. Summary of the relevant findings for each selection method.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Cite this article.

Kurysheva, A., van Rijen, H.V.M., Stolte, C. et al. Validity, acceptability, and procedural issues of selection methods for graduate study admissions in the fields of science, technology, engineering, and mathematics: a mapping review. IJ STEM Ed 10 , 55 (2023). https://doi.org/10.1186/s40594-023-00445-4

Download citation

Received : 10 January 2023

Accepted : 02 August 2023

Published : 07 September 2023

DOI : https://doi.org/10.1186/s40594-023-00445-4

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Graduate admissions
• Selection methods
• Predictive validity

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

• View all journals
• Explore content
• About the journal
• Publish with us
• Sign up for alerts
• 12 March 2024

Bring PhD assessment into the twenty-first century

You have full access to this article via your institution.

Innovation in PhD education has not reached how doctoral degrees are assessed. Credit: Dan Dunkley/Science Photo Library

Research and teaching in today’s universities are unrecognizable compared with what they were in the early nineteenth century, when Germany and later France gave the world the modern research doctorate. And yet significant aspects of the process of acquiring and assessing a doctorate have remained remarkably constant. A minimum of three years of independent study mentored by a single individual culminates in the production of the doctoral thesis — often a magisterial, book-length piece of work that is assessed in an oral examination by a few senior academic researchers. In an age in which there is much research-informed innovation in teaching and learning, the assessment of the doctoral thesis represents a curious throwback that is seemingly impervious to meaningful reform.

But reform is needed. Some doctoral candidates perceive the current assessment system to lack transparency, and examiners report concerns of falling standards ( G. Houston A Study of the PhD Examination: Process, Attributes and Outcomes . PhD thesis, Oxford Univ.; 2018 ). Making the qualification more structured would help — and, equally importantly, would bring the assessment of PhD education in line with education across the board. PhD candidates with experience of modern assessment methods will become better researchers, wherever they work. Indeed, most will not be working in universities: the majority of PhD holders find employment outside academia.

Collection: Career resources for PhD students

It’s not that PhD training is completely stuck in the nineteenth century. Today’s doctoral candidates can choose from a range of pathways. Professional doctorates, often used in engineering, are jointly supervised by an employer and an academic, and are aimed at solving industry-based problems. Another innovation is PhD by publication, in which, instead of a final thesis on one or more research questions, the criterion for an award is a minimum number of papers published or accepted for publication. In some countries, doctoral students are increasingly being trained in cohorts, with the aim of providing a less isolating experience than that offered by the conventional supervisor–student relationship. PhD candidates are also encouraged to acquire transferable skills — for example, in data analysis, public engagement, project management or business, economics and finance. The value of such training would be even greater if these skills were to be formally assessed alongside a dissertation rather than seen as optional.

And yet, most PhDs are still assessed after the production of a final dissertation, according to a format that, at its core, has not changed for at least half a century, as speakers and delegates noted at an event in London last month on PhD assessment, organized by the Society for Research in Higher Educatio n. Innovations in assessment that are common at other levels of education are struggling to find their way into the conventional doctoral programme.

Take the concept of learning objectives. Intended to aid consistency, fairness and transparency, learning objectives are a summary of what a student is expected to know and how they will be assessed, and are given at the start of a course of study. Part of the ambition is also to help tutors to keep track of their students’ learning and take remedial action before it is too late.

PhD training is no longer fit for purpose — it needs reform now

Formative assessment is another practice that has yet to find its way into PhD assessment consistently. Here, a tutor evaluates a student’s progress at the mid-point of a course and gives feedback or guidance on what students need to do to improve ahead of their final, or summative, assessment. It is not that these methods are absent from modern PhDs; a conscientious supervisor will not leave candidates to sink or swim until the last day. But at many institutions, such approaches are not required of PhD supervisors.

Part of the difficulty is that PhD training is carried out in research departments by people who do not need to have teaching qualifications or awareness of innovations based on education research. Supervisors shouldn’t just be experts in their field, they should also know how best to convey that subject knowledge — along with knowledge of research methods — to their students.

It is probably not possible for universities to require all doctoral supervisors to have teaching qualifications. But there are smaller changes that can be made. At a minimum, doctoral supervisors should take the time to engage with the research that exists in the field of PhD education, and how it can apply to their interactions with students.

There can be no one-size-fits-all solution to improving how a PhD is assessed, because different subjects often have bespoke needs and practices ( P. Denicolo Qual. Assur. Educ. 11 , 84–91; 2003 ). But supervisors and representatives of individual subject communities must continue to discuss what is most appropriate for their disciplines.

All things considered, there is benefit to adopting a more structured approach to PhD assessment. It is high time that PhD education caught up with changes that are now mainstream at most other levels of education. That must start with a closer partnership between education researchers, PhD supervisors and organizers of doctoral-training programmes in universities. This partnership will benefit everyone — PhD supervisors and doctoral students coming into the research workforce, whether in universities or elsewhere.

Education and training in research has entered many secondary schools, along with undergraduate teaching, which is a good thing. In the spirit of mutual learning, research doctoral supervisors, too, will benefit by going back to school.

Nature 627 , 244 (2024)

doi: https://doi.org/10.1038/d41586-024-00718-0

Reprints and permissions

Related Articles

• Scientific community

Four years on: the career costs for scientists battling long COVID

Career Feature 18 MAR 24

People, passion, publishable: an early-career researcher’s checklist for prioritizing projects

Career Column 15 MAR 24

Divas, captains, ghosts, ants and bumble-bees: collaborator attitudes explained

How to stop 'passing the harasser': universities urged to join information-sharing scheme

News 18 MAR 24

Take these steps to accelerate the path to gender equity in health sciences

Nature Index 13 MAR 24

Pay for trees with carbon credits to deliver urban green spaces for all

Correspondence 12 MAR 24

Backend/ DevOps Engineer – National Facility for Data Handling and Analysis

APPLICATION CLOSING DATE: April 2nd, 2024. About the institute Human Technopole (HT) is a new interdisciplinary life science research institute, cr...

Human Technopole

Professorship for Pneumology (W3)

The Jena University Hospital (JUH) invites applications for a Professorship for Pneumology (W3) to be filled at the earliest possible date. The pro...

07743 Jena, Thüringen (DE)

Friedrich-Schiller-Universität Jena

Faculty Positions at Great Bay University, China

We are now seeking outstanding candidates in Physics, Chemistry and Physical Sciences.

Dongguan, Guangdong, China

Great Bay University, China (GBU)

Tenure-Track Assistant Professor to the rank of Associate Professor in computational biology

UNIL is a leading international teaching and research institution, with over 5,000 employees and 15,500 students split between its Dorigny campus, ...

Lausanne, Canton of Vaud (CH)

University of Lausanne (UNIL)

Assistant Scientist/Professor in Rare Disease Research, Sanford Research

Assistant Scientist/Professor in Rare Disease Research, Sanford Research Sanford Research invites applications for full-time faculty at the rank of...

Sioux Falls, South Dakota

Sanford Research

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Quick links

• Explore articles by subject
• Guide to authors
• Editorial policies

Call us : 0091 80 4111 6961

Crafting Effective Questionnaires for PhD Research: A Step-by-Step Guide

• PhD Research

Do you know the major problems researchers can face if they don’t craft productive PhD research questionnaires ? They may be unable to replicate the research and are also unable to help the readers understand the answers of the research questions. And not only that, but crafting ineffective questionnaires for your PhD research, can lead to your entire research being a futile prospect. But the story takes a turn.

After extensive research, we have understood that there are basically 3 steps to craft effective questionnaires for your PhD research. In this blog, we are going to describe those 3 steps so that you not only craft effective questionnaires but also help others to craft Effective Questionnaires for your PhD research. So, let’s get started, shall we?

But wait 🤚!!! Do these three methods help you create good surveys for your PhD research? is the first query you ought to address to yourself. I mean, is there a crucial query you ought to have answered before diving into the subject? Please think through and then read the remaining blog.

Why is it necessary to design efficient questionnaires for PhD research? So you might not be able to create the ideal questionnaire for your PhD if you don’t know the reason. As a result, you could be asking, “What is the solution?” Please read the remaining posts on the blog to learn more about this.

Crafting effective questionnaires is crucial for PhD research for several reasons:

• Obtaining reliable and valid data : Effective questionnaires ensure that the data collected is reliable and valid, which is essential for making accurate conclusions and recommendations based on the research findings.
• Enhancing the credibility of the research : If a questionnaire is poorly constructed, it can undermine the credibility of the research and make it difficult to convince others of the findings.
• Improving response rates : An effective questionnaire is more likely to be completed by respondents, resulting in higher response rates and more representative data.
• Reducing bias : A well-crafted questionnaire reduces the potential for bias in the responses by ensuring that questions are clear, unbiased, and focused on the research objectives.
• Saving time and resources : By ensuring that the questionnaire is well-designed, researchers can save time and resources by collecting data that is directly relevant to the research question.
• Facilitating data analysis : An effective questionnaire can make data analysis easier and more accurate by ensuring that the questions are structured in a logical and consistent manner.

Hence, crafting an effective questionnaire is essential for obtaining reliable and valid data, enhancing the credibility of the research, improving response rates, reducing bias, saving time and resources, and facilitating data analysis. So, let’s jump into knowing the answers to these questions.

PhD research questionnaires development and validation

Before moving with this part, we have something important to discuss regarding the development of the PhD research questions. Can you guess what? It is as important as knowing the development process of PhD research questions. 

Developing effective research questions is an essential step in the process of conducting a PhD research project. Here are some tips to help you develop effective PhD research questions:

• Start with a broad topic : Begin by identifying a broad topic area that you are interested in and that has not been extensively researched. The topic should be significant and relevant to your field of study.
• Review existing literature : Conduct a thorough review of existing literature to identify research gaps and potential areas of exploration.
• Narrow down your focus : Once you have identified a research gap, narrow down your focus by formulating research questions that are specific, focused, and clear. Avoid broad and vague questions that are difficult to answer.
• Make sure your research questions are feasible : Your research questions should be feasible and answerable within the timeframe and resources available for your PhD project.
• Test your questions : Share your research questions with your supervisor and peers to get feedback and refine them further.
• Make sure your research questions are original : Ensure that your research questions are original and contribute to the existing body of knowledge in your field.
• Revise and refine : Continuously revise and refine your research questions throughout the PhD project as you gain more knowledge and insights.

Remember that developing effective PhD research questions is an iterative process and requires time, effort, and collaboration with your supervisor and peers. 

Now, another question can come in our mind which is “why validation is needed for PhD research questionnaires?” It will help you decide whether to validate the questionnaires or not. So, let us know the answer to this question and then decide.

Validation is essential for PhD research questionnaires for several reasons:

• Ensuring reliability : Validation helps ensure that the questionnaire measures what it is intended to measure consistently across different participants and situations. This increases the validity of the data that is gathered.
• Minimizing measurement errors : Validation helps identify and minimize measurement errors that could lead to inaccurate data and potentially flawed research conclusions.
• Increasing validity : Validation helps ensure that the questionnaire is measuring the construct or concept it is intended to measure. This increases the validity of the data collected and the research conclusions.
• Enhancing credibility : A validated questionnaire enhances the credibility of the research and can make it easier to convince others of the research findings.
• Improving research quality : A validated questionnaire can lead to better quality research by ensuring that the data collected is relevant, reliable, and valid.
• Meeting ethical standards : Validating the questionnaire helps ensure that participants are not subjected to unnecessary or irrelevant questions, which is important for meeting ethical standards in research.

Hence, validation is needed for PhD research questionnaires to ensure reliability, minimize measurement errors, increase validity, enhance credibility, improve research quality, and meet ethical standards.

Validating a PhD research questionnaire involves several steps. Here are some key steps to consider:

• Develop a clear research question : The first step in validating a questionnaire is to develop a clear research question that the questionnaire is designed to answer.
• Determine the type of validity: There are different types of validity that a questionnaire can have, such as content validity, construct validity, criterion-related validity, and face validity. Determine which type(s) of validity are most relevant to your research.
• Develop the questionnaire: Develop the questionnaire based on the research question and the type(s) of validity being assessed. Ensure that the questions are clear, unbiased, and relevant to the research objectives.
• Conduct a pilot study : Administer the questionnaire to a small sample of participants (e.g., 10-15) to identify any problems with the questionnaire and assess the validity of the questions.
• Evaluate the questionnaire : Evaluate the questionnaire for content validity, construct validity, criterion-related validity, and face validity based on the data collected from the pilot study.
• Refine the questionnaire : Refine the questionnaire based on the feedback received during the pilot study and the validity assessment.
• Administer the questionnaire : Administer the final version of the questionnaire to the target population.
• Analyze the data : Analyze the data collected from the questionnaire to determine the reliability and validity of the questionnaire.
• Report the results : Report the results of the validity assessment in the research report, including the methods used to assess validity, the results of the assessment, and any limitations of the questionnaire.

Hence, validating a PhD research questionnaire involves developing a clear research question, determining the type(s) of validity to be assessed, developing the questionnaire, conducting a pilot study, evaluating the questionnaire, refining the questionnaire, administering the questionnaire, analyzing the data, and reporting the results.

Now, it’s time to go to the 2nd step which can help you a little more in crafting better questions in PhD research.  

Types of validation of PhD research questionnaires

Now, it’s time to understand the different types of validation of the PhD research questionnaire. But again , the questioning will not end. Why do we need to know about different types of validation of PhD research questionnaires? 

Knowing about different types of validation of PhD research questionnaires is important for several reasons:

• Ensuring the reliability and validity of data : Different types of validation can help ensure that the data collected from the questionnaire is reliable and valid, which is essential for making accurate conclusions and recommendations based on the research findings.
• Selecting the appropriate type of validation : Depending on the research question and the type of data being collected, different types of validation may be more appropriate. Knowing about different types of validation can help researchers select the most appropriate type(s) of validation for their research.
• Enhancing the credibility of the research : A well-validated questionnaire enhances the credibility of the research and can make it easier to convince others of the research findings.
• Improving research quality : Validating the questionnaire can lead to better quality research by ensuring that the data collected is relevant, reliable, and valid.

Now, I think there is no question left in this part except knowing the types of validation of PhD research questionnaires. If you have any questions in your mind, then you can comment below so that we can update the blog. So, let us jump into the answer to this question.

There are several types of validation of PhD research questionnaires. Some of the most typical varieties are listed below:

• Content validity : Content validity refers to the extent to which the questionnaire items adequately cover the intended content area. To assess content validity, researchers typically seek input from subject matter experts or use established guidelines or criteria to evaluate the relevance of the questionnaire items.
• Construct validity : Construct validity refers to the extent to which the questionnaire items measure the intended construct or concept. To assess construct validity, researchers may use statistical techniques, such as factor analysis or confirmatory factor analysis, to examine how well the questionnaire items align with the underlying construct.
• Criterion-related validity : Criterion-related validity refers to the extent to which the questionnaire items are related to an external criterion or standard that is known to be related to the construct of interest. To assess criterion-related validity, researchers may compare the questionnaire scores to scores on a standardized test or other measures of the same construct.
• Face validity : Face validity refers to the extent to which the questionnaire items appear to be relevant and appropriate to the participants. To assess face validity, researchers may ask participants to review the questionnaire and provide feedback on the clarity, relevance, and appropriateness of the items.
• Concurrent validity : Concurrent validity refers to the extent to which the questionnaire items correlate with an external criterion measured at the same time. For example, if a questionnaire is designed to measure depression, researchers may compare the questionnaire scores to scores on a depression scale administered at the same time.
• Predictive validity : Predictive validity refers to the extent to which the questionnaire items predict future behaviour or outcomes related to the construct of interest. For example, if a questionnaire is designed to measure job satisfaction, researchers may use the questionnaire scores to predict future job performance or turnover.

Hence, the most common types of validation of PhD research questionnaires include content validity, construct validity, criterion-related validity, face validity, concurrent validity, and predictive validity.

Principles and methods of PhD research questionnaires

We will divide this blog into two parts, in one part, we will describe the principles of PhD research questionnaires and in the next part, we will describe the methods of PhD research questionnaires. So, let us start the blog with the first part.

Understanding the principles of PhD research questionnaires is important because it enables a researcher to design effective and relevant questionnaires for their research. By following these principles, the researcher can ensure that the questions are clear, relevant, specific, feasible, original, testable, and significant, which will help them to gather accurate and useful data to answer their research questions. 

Additionally, understanding the methods of designing and administering research questionnaires will help the researcher to avoid common pitfalls and mistakes in the process, such as asking biased or leading questions, administering the questionnaire to an inappropriate population, or failing to pilot test the questionnaire. Ultimately, a well-designed research questionnaire can be a valuable tool for gathering data in a PhD research project and can contribute to the development of new knowledge in the researcher’s field of study. 

When formulating research questions for a PhD project, there are several principles that you should keep in mind:

• Clarity : Your research questions should be clear and concise so that readers can easily understand what you are investigating.
• Relevance : Your research questions should be relevant to your field of study and contribute to the existing body of knowledge.
• Specificity : Your research questions should be specific enough to guide your research and help you to focus on the key issues that you want to explore.
• Feasibility : Your research questions should be feasible to answer given the resources and time available for your PhD project.
• Originality : Your research questions should be original and innovative so that they contribute to the development of new knowledge in your field.
• Testability : Your research questions should be testable through empirical research methods so that you can gather data to support or refute your hypotheses.
• Significance : Your research questions should be significant in terms of their potential impact on your field of study, and should address important research gaps or unanswered questions.

By following these principles, you can develop research questions that will guide your PhD project and contribute to the advancement of knowledge in your field.

Now, it’s time to know the second part of this question which is the methods of PhD research questionnaires. It is the last step for us to craft better questionnaires for PhD research. 

Research questionnaires can be a useful tool for gathering data in a PhD research project. When designing a research questionnaire, you should consider the following methods:

• Identify the research questions : The first step is to identify the research questions that you want to answer. Your questionnaire should be designed to collect data that will help you to answer these questions.
• Choose the appropriate type of questions : Decide on the type of questions you will use, such as open-ended or closed-ended questions. Closed-ended questions are usually easier to analyze and quantify, while open-ended questions can provide more in-depth and nuanced responses.
• Determine the format of the questionnaire : The questionnaire can be administered online or in person, and can be structured or unstructured. The format will depend on the nature of your research questions and the target population.
• Develop the questions : Develop clear and concise questions that are easy to understand and answer. Avoid using jargon or technical language that may be unfamiliar to your respondents.
• Pilot tests the questionnaire : Before administering the questionnaire to your target population, conduct a pilot test with a small group of people to identify any potential issues or misunderstandings.
• Administer the questionnaire : Once the questionnaire is finalized, administer it to your target population. You may need to provide instructions or assistance to ensure that respondents understand the questions and how to answer them.
• Analyze the data : After collecting the data, analyze it using statistical or qualitative methods, depending on the nature of the data and research questions.

By using these methods, you can develop an effective research questionnaire that will help you to collect data and answer your research questions.

But wait!!! It’s not over yet. I hope you are a research enthusiast who wants to know more about creating better PhD research questions . Also, if you want us to help you in this matter, you can definitely contact us with the given contact information on the website. 

We haven’t answered one question in this blog. Can you guess the question? Then tell us in the comments.

Leave a Reply Cancel reply

Your email address will not be published. Required fields are marked *

Request a Call Back

• Academic Writing
• Avoiding Rejection
• Data Analysis
• Data collection
• Dissertation
• Dissertation Defense Preparation
• Experimental Research
• Limitation and Delimitation
• Literature Review
• Manuscript Publication
• Quantitative Research
• Questionnaires
• Research Design
• Research Methodology
• Research Paper Writing
• Research Proposal
• Research Scholar
• Topic Selection
• Uncategorized

Recent Posts

• A Beginner’s Guide to Top 3 Best PhD Research Topics for 2024-25 thesis , Topic Selection January 16, 2024
• How to Write a Research Proposal and How NOT to in 2024 Research Proposal December 20, 2023
• 5 Unknown Differences Between Limitation and Delimitation Limitation and Delimitation November 21, 2023
• 3 Game-Changing Tips for PhD Thesis Writing in 2023-24 Citation , PhD Research , PhD Thesis November 3, 2023
• What to Do When PhD Dissertation Defense Preparation Derail? Dissertation Defense Preparation September 16, 2023

How to get published in SCI Indexed Journals