further research recommendations

Have a language expert improve your writing

Run a free plagiarism check in 10 minutes, generate accurate citations for free.

• Knowledge Base
• Research paper
• How to Write Recommendations in Research | Examples & Tips

How to Write Recommendations in Research | Examples & Tips

Published on September 15, 2022 by Tegan George . Revised on July 18, 2023.

Recommendations in research are a crucial component of your discussion section and the conclusion of your thesis , dissertation , or research paper .

As you conduct your research and analyze the data you collected , perhaps there are ideas or results that don’t quite fit the scope of your research topic. Or, maybe your results suggest that there are further implications of your results or the causal relationships between previously-studied variables than covered in extant research.

Instantly correct all language mistakes in your text

Upload your document to correct all your mistakes in minutes

Table of contents

What should recommendations look like, building your research recommendation, how should your recommendations be written, recommendation in research example, other interesting articles, frequently asked questions about recommendations.

Recommendations for future research should be:

• Concrete and specific
• Supported with a clear rationale
• Directly connected to your research

Overall, strive to highlight ways other researchers can reproduce or replicate your results to draw further conclusions, and suggest different directions that future research can take, if applicable.

Relatedly, when making these recommendations, avoid:

• Undermining your own work, but rather offer suggestions on how future studies can build upon it
• Suggesting recommendations actually needed to complete your argument, but rather ensure that your research stands alone on its own merits
• Using recommendations as a place for self-criticism, but rather as a natural extension point for your work

Scribbr Citation Checker New

The AI-powered Citation Checker helps you avoid common mistakes such as:

• Missing commas and periods
• Incorrect usage of “et al.”
• Ampersands (&) in narrative citations
• Missing reference entries

There are many different ways to frame recommendations, but the easiest is perhaps to follow the formula of research question   conclusion  recommendation. Here’s an example.

Conclusion An important condition for controlling many social skills is mastering language. If children have a better command of language, they can express themselves better and are better able to understand their peers. Opportunities to practice social skills are thus dependent on the development of language skills.

As a rule of thumb, try to limit yourself to only the most relevant future recommendations: ones that stem directly from your work. While you can have multiple recommendations for each research conclusion, it is also acceptable to have one recommendation that is connected to more than one conclusion.

These recommendations should be targeted at your audience, specifically toward peers or colleagues in your field that work on similar subjects to your paper or dissertation topic . They can flow directly from any limitations you found while conducting your work, offering concrete and actionable possibilities for how future research can build on anything that your own work was unable to address at the time of your writing.

See below for a full research recommendation example that you can use as a template to write your own.

The only proofreading tool specialized in correcting academic writing - try for free!

The academic proofreading tool has been trained on 1000s of academic texts and by native English editors. Making it the most accurate and reliable proofreading tool for students.

Try for free

If you want to know more about AI for academic writing, AI tools, or research bias, make sure to check out some of our other articles with explanations and examples or go directly to our tools!

Research bias

• Survivorship bias
• Self-serving bias
• Availability heuristic
• Halo effect
• Hindsight bias
• Deep learning
• Generative AI
• Machine learning
• Reinforcement learning
• Supervised vs. unsupervised learning

 (AI) Tools

• Grammar Checker
• Paraphrasing Tool
• Text Summarizer
• AI Detector
• Plagiarism Checker
• Citation Generator

While it may be tempting to present new arguments or evidence in your thesis or disseration conclusion , especially if you have a particularly striking argument you’d like to finish your analysis with, you shouldn’t. Theses and dissertations follow a more formal structure than this.

All your findings and arguments should be presented in the body of the text (more specifically in the discussion section and results section .) The conclusion is meant to summarize and reflect on the evidence and arguments you have already presented, not introduce new ones.

The conclusion of your thesis or dissertation should include the following:

• A restatement of your research question
• A summary of your key arguments and/or results
• A short discussion of the implications of your research

For a stronger dissertation conclusion , avoid including:

• Important evidence or analysis that wasn’t mentioned in the discussion section and results section
• Generic concluding phrases (e.g. “In conclusion …”)
• Weak statements that undermine your argument (e.g., “There are good points on both sides of this issue.”)

Your conclusion should leave the reader with a strong, decisive impression of your work.

In a thesis or dissertation, the discussion is an in-depth exploration of the results, going into detail about the meaning of your findings and citing relevant sources to put them in context.

The conclusion is more shorter and more general: it concisely answers your main research question and makes recommendations based on your overall findings.

Cite this Scribbr article

If you want to cite this source, you can copy and paste the citation or click the “Cite this Scribbr article” button to automatically add the citation to our free Citation Generator.

George, T. (2023, July 18). How to Write Recommendations in Research | Examples & Tips. Scribbr. Retrieved March 25, 2024, from https://www.scribbr.com/dissertation/recommendations-in-research/

Is this article helpful?

Tegan George

Other students also liked, how to write a discussion section | tips & examples, how to write a thesis or dissertation conclusion, how to write a results section | tips & examples, "i thought ai proofreading was useless but..".

I've been using Scribbr for years now and I know it's a service that won't disappoint. It does a good job spotting mistakes”

• Privacy Policy

Buy Me a Coffee

Home » Research Recommendations – Examples and Writing Guide

Research Recommendations – Examples and Writing Guide

Table of Contents

Research Recommendations

Definition:

Research recommendations refer to suggestions or advice given to someone who is looking to conduct research on a specific topic or area. These recommendations may include suggestions for research methods, data collection techniques, sources of information, and other factors that can help to ensure that the research is conducted in a rigorous and effective manner. Research recommendations may be provided by experts in the field, such as professors, researchers, or consultants, and are intended to help guide the researcher towards the most appropriate and effective approach to their research project.

Parts of Research Recommendations

Research recommendations can vary depending on the specific project or area of research, but typically they will include some or all of the following parts:

• Research question or objective : This is the overarching goal or purpose of the research project.
• Research methods : This includes the specific techniques and strategies that will be used to collect and analyze data. The methods will depend on the research question and the type of data being collected.
• Data collection: This refers to the process of gathering information or data that will be used to answer the research question. This can involve a range of different methods, including surveys, interviews, observations, or experiments.
• Data analysis : This involves the process of examining and interpreting the data that has been collected. This can involve statistical analysis, qualitative analysis, or a combination of both.
• Results and conclusions: This section summarizes the findings of the research and presents any conclusions or recommendations based on those findings.
• Limitations and future research: This section discusses any limitations of the study and suggests areas for future research that could build on the findings of the current project.

How to Write Research Recommendations

Writing research recommendations involves providing specific suggestions or advice to a researcher on how to conduct their study. Here are some steps to consider when writing research recommendations:

• Understand the research question: Before writing research recommendations, it is important to have a clear understanding of the research question and the objectives of the study. This will help to ensure that the recommendations are relevant and appropriate.
• Consider the research methods: Consider the most appropriate research methods that could be used to collect and analyze data that will address the research question. Identify the strengths and weaknesses of the different methods and how they might apply to the specific research question.
• Provide specific recommendations: Provide specific and actionable recommendations that the researcher can implement in their study. This can include recommendations related to sample size, data collection techniques, research instruments, data analysis methods, or other relevant factors.
• Justify recommendations : Justify why each recommendation is being made and how it will help to address the research question or objective. It is important to provide a clear rationale for each recommendation to help the researcher understand why it is important.
• Consider limitations and ethical considerations : Consider any limitations or potential ethical considerations that may arise in conducting the research. Provide recommendations for addressing these issues or mitigating their impact.
• Summarize recommendations: Provide a summary of the recommendations at the end of the report or document, highlighting the most important points and emphasizing how the recommendations will contribute to the overall success of the research project.

Example of Research Recommendations

Example of Research Recommendations sample for students:

• Further investigate the effects of X on Y by conducting a larger-scale randomized controlled trial with a diverse population.
• Explore the relationship between A and B by conducting qualitative interviews with individuals who have experience with both.
• Investigate the long-term effects of intervention C by conducting a follow-up study with participants one year after completion.
• Examine the effectiveness of intervention D in a real-world setting by conducting a field study in a naturalistic environment.
• Compare and contrast the results of this study with those of previous research on the same topic to identify any discrepancies or inconsistencies in the findings.
• Expand upon the limitations of this study by addressing potential confounding variables and conducting further analyses to control for them.
• Investigate the relationship between E and F by conducting a meta-analysis of existing literature on the topic.
• Explore the potential moderating effects of variable G on the relationship between H and I by conducting subgroup analyses.
• Identify potential areas for future research based on the gaps in current literature and the findings of this study.
• Conduct a replication study to validate the results of this study and further establish the generalizability of the findings.

Applications of Research Recommendations

Research recommendations are important as they provide guidance on how to improve or solve a problem. The applications of research recommendations are numerous and can be used in various fields. Some of the applications of research recommendations include:

• Policy-making: Research recommendations can be used to develop policies that address specific issues. For example, recommendations from research on climate change can be used to develop policies that reduce carbon emissions and promote sustainability.
• Program development: Research recommendations can guide the development of programs that address specific issues. For example, recommendations from research on education can be used to develop programs that improve student achievement.
• Product development : Research recommendations can guide the development of products that meet specific needs. For example, recommendations from research on consumer behavior can be used to develop products that appeal to consumers.
• Marketing strategies: Research recommendations can be used to develop effective marketing strategies. For example, recommendations from research on target audiences can be used to develop marketing strategies that effectively reach specific demographic groups.
• Medical practice : Research recommendations can guide medical practitioners in providing the best possible care to patients. For example, recommendations from research on treatments for specific conditions can be used to improve patient outcomes.
• Scientific research: Research recommendations can guide future research in a specific field. For example, recommendations from research on a specific disease can be used to guide future research on treatments and cures for that disease.

Purpose of Research Recommendations

The purpose of research recommendations is to provide guidance on how to improve or solve a problem based on the findings of research. Research recommendations are typically made at the end of a research study and are based on the conclusions drawn from the research data. The purpose of research recommendations is to provide actionable advice to individuals or organizations that can help them make informed decisions, develop effective strategies, or implement changes that address the issues identified in the research.

The main purpose of research recommendations is to facilitate the transfer of knowledge from researchers to practitioners, policymakers, or other stakeholders who can benefit from the research findings. Recommendations can help bridge the gap between research and practice by providing specific actions that can be taken based on the research results. By providing clear and actionable recommendations, researchers can help ensure that their findings are put into practice, leading to improvements in various fields, such as healthcare, education, business, and public policy.

Characteristics of Research Recommendations

Research recommendations are a key component of research studies and are intended to provide practical guidance on how to apply research findings to real-world problems. The following are some of the key characteristics of research recommendations:

• Actionable : Research recommendations should be specific and actionable, providing clear guidance on what actions should be taken to address the problem identified in the research.
• Evidence-based: Research recommendations should be based on the findings of the research study, supported by the data collected and analyzed.
• Contextual: Research recommendations should be tailored to the specific context in which they will be implemented, taking into account the unique circumstances and constraints of the situation.
• Feasible : Research recommendations should be realistic and feasible, taking into account the available resources, time constraints, and other factors that may impact their implementation.
• Prioritized: Research recommendations should be prioritized based on their potential impact and feasibility, with the most important recommendations given the highest priority.
• Communicated effectively: Research recommendations should be communicated clearly and effectively, using language that is understandable to the target audience.
• Evaluated : Research recommendations should be evaluated to determine their effectiveness in addressing the problem identified in the research, and to identify opportunities for improvement.

Advantages of Research Recommendations

Research recommendations have several advantages, including:

• Providing practical guidance: Research recommendations provide practical guidance on how to apply research findings to real-world problems, helping to bridge the gap between research and practice.
• Improving decision-making: Research recommendations help decision-makers make informed decisions based on the findings of research, leading to better outcomes and improved performance.
• Enhancing accountability : Research recommendations can help enhance accountability by providing clear guidance on what actions should be taken, and by providing a basis for evaluating progress and outcomes.
• Informing policy development : Research recommendations can inform the development of policies that are evidence-based and tailored to the specific needs of a given situation.
• Enhancing knowledge transfer: Research recommendations help facilitate the transfer of knowledge from researchers to practitioners, policymakers, or other stakeholders who can benefit from the research findings.
• Encouraging further research : Research recommendations can help identify gaps in knowledge and areas for further research, encouraging continued exploration and discovery.
• Promoting innovation: Research recommendations can help identify innovative solutions to complex problems, leading to new ideas and approaches.

Limitations of Research Recommendations

While research recommendations have several advantages, there are also some limitations to consider. These limitations include:

• Context-specific: Research recommendations may be context-specific and may not be applicable in all situations. Recommendations developed in one context may not be suitable for another context, requiring adaptation or modification.
• I mplementation challenges: Implementation of research recommendations may face challenges, such as lack of resources, resistance to change, or lack of buy-in from stakeholders.
• Limited scope: Research recommendations may be limited in scope, focusing only on a specific issue or aspect of a problem, while other important factors may be overlooked.
• Uncertainty : Research recommendations may be uncertain, particularly when the research findings are inconclusive or when the recommendations are based on limited data.
• Bias : Research recommendations may be influenced by researcher bias or conflicts of interest, leading to recommendations that are not in the best interests of stakeholders.
• Timing : Research recommendations may be time-sensitive, requiring timely action to be effective. Delayed action may result in missed opportunities or reduced effectiveness.
• Lack of evaluation: Research recommendations may not be evaluated to determine their effectiveness or impact, making it difficult to assess whether they are successful or not.

About the author

Muhammad Hassan

Researcher, Academic Writer, Web developer

You may also like

Research Paper Conclusion – Writing Guide and...

Appendices – Writing Guide, Types and Examples

Research Report – Example, Writing Guide and...

Delimitations in Research – Types, Examples and...

Scope of the Research – Writing Guide and...

Research Contribution – Thesis Guide

Educational resources and simple solutions for your research journey

What are Implications and Recommendations in Research? How to Write It, with Examples

Highly cited research articles often contain both implications and recommendations , but there is often some confusion around the difference between implications and recommendations in research. Implications of a study are the impact your research makes in your chosen area; they discuss how the findings of the study may be important to justify further exploration of your research topic. Research recommendations suggest future actions or subsequent steps supported by your research findings. It helps to improve your field of research or cross-disciplinary fields through future research or provides frameworks for decision-makers or policymakers. Recommendations are the action plan you propose based on the outcome.

In this article, we aim to simplify these concepts for researchers by providing key insights on the following:  

• what are implications in research 
• what is recommendation in research 
• differences between implications and recommendations 
• how to write implications in research 
• how to write recommendation in research 
• sample recommendation in research 

Table of Contents

What are implications in research

The implications in research explain what the findings of the study mean to researchers or to certain subgroups or populations beyond the basic interpretation of results. Even if your findings fail to bring radical or disruptive changes to existing ways of doing things, they might have important implications for future research studies. For example, your proposed method for operating remote-controlled robots could be more precise, efficient, or cheaper than existing methods, or the remote-controlled robot could be used in other application areas. This could enable more researchers to study a specific problem or open up new research opportunities.   

Implications in research inform how the findings, drawn from your results, may be important for and impact policy, practice, theory, and subsequent research. Implications may be theoretical or practical. 1  

• Practical implications are potential values of the study with practical or real outcomes . Determining the practical implications of several solutions can aid in identifying optimal solution results. For example, clinical research or research on classroom learning mostly has practical implications in research . If you developed a new teaching method, the implication would be how teachers can use that method based on your findings.  
• Theoretical implications in research constitute additions to existing theories or establish new theories. These types of implications in research characterize the ability of research to influence society in apparent ways. It is, at most, an educated guess (theoretical) about the possible implication of action and need not be as absolute as practical implications in research . If your study supported the tested theory, the theoretical implication would be that the theory can explain the investigated phenomenon. Else, your study may serve as a basis for modifying the theory. Theories may be partially supported as well, implying further study of the theory or necessary modifications are required.  

What are recommendations in research?

Recommendations in research can be considered an important segment of the analysis phase. Recommendations allow you to suggest specific interventions or strategies to address the issues and constraints identified through your study. It responds to key findings arrived at through data collection and analysis. A process of prioritization can help you narrow down important findings for which recommendations are developed.  

Recommendations in research examples

Recommendations in research may vary depending on the purpose or beneficiary as seen in the table below.  

Table: Recommendations in research examples based on purpose and beneficiary  

If you’re wondering how to make recommendations in research . You can use the simple  recommendation in research example below as a handy template.  

Table: Sample recommendation in research template  

Basic differences between implications and recommendations in research

Implications and recommendations in research are two important aspects of a research paper or your thesis or dissertation. Implications discuss the importance of the research findings, while recommendations offer specific actions to solve a problem. So, the basic difference between the two is in their function and the questions asked to achieve it. The following table highlights the main differences between implications and recommendations in research .  

Table: Differences between implications and recommendations in research  

Where do implications go in your research paper.

Because the implications and recommendations of the research are based on study findings, both are usually written after the completion of a study. There is no specific section dedicated to implications in research ; they are usually integrated into the discussion section adding evidence as to why the results are meaningful and what they add to the field. Implications can be written after summarizing your main findings and before the recommendations and conclusion.   

Implications can also be presented in the conclusion section after a short summary of the study results.   

How to write implications in research

Implication means something that is inferred. The implications of your research are derived from the importance of your work and how it will impact future research. It is based on how previous studies have advanced your field and how your study can add to that.   

When figuring out how to write implications in research , a good strategy is to separate it into the different types of implications in research , such as social, political, technological, policy-related, or others. As mentioned earlier, the most frequently used are the theoretical and practical implications.   

Next, you need to ask, “Who will benefit the most from reading my paper?” Is it policymakers, physicians, the public, or other researchers? Once you know your target population, explain how your findings can help them.  

The implication section can include a paragraph or two that asserts the practical or managerial implications and links it to the study findings. A discussion can then follow, demonstrating that the findings can be practically implemented or how they will benefit a specific audience. The writer is given a specific degree of freedom when writing research implications , depending on the type of implication in research you want to discuss: practical or theoretical. Each is discussed differently, using different words or in separate sections. The implications can be based on how the findings in your study are similar or dissimilar to that in previous studies. Your study may reaffirm or disprove the results of other studies, which has important implications in research . You can also suggest future research directions in the light of your findings or require further research to confirm your findings, which are all crucial implications. Most importantly, ensure the implications in research are specific and that your tone reflects the strength of your findings without exaggerating your results.   

Implications in research can begin with the following specific sentence structures:  

• These findings suggest that…
• These results build on existing body of evidence of…
• These results should be considered when…
• While previous research focused on x, our results show that y…

What should recommendations in research look like?

Recommendations for future research should be:  

• Directly related to your research question or findings  
• Concrete and specific  
• Supported by a clear reasoning  

The recommendations in research can be based on the following factors:  

1. Beneficiary: A paper’s research contribution may be aimed at single or multiple beneficiaries, based on which recommendations can vary. For instance, if your research is about the quality of care in hospitals, the research recommendation to different beneficiaries might be as follows:  

• Nursing staff: Staff should undergo training to enhance their understanding of what quality of care entails.  
• Health science educators: Educators must design training modules that address quality-related issues in the hospital.  
• Hospital management: Develop policies that will increase staff participation in training related to health science.  

2. Limitations: The best way to figure out what to include in your research recommendations is to understand the limitations of your study. It could be based on factors that you have overlooked or could not consider in your present study. Accordingly, the researcher can recommend that other researchers approach the problem from a different perspective, dimension, or methodology. For example, research into the quality of care in hospitals can be based on quantitative data. The researcher can then recommend a qualitative study of factors influencing the quality of care, or they can suggest investigating the problem from the perspective of patients rather than the healthcare providers.   

3. Theory or Practice: Your recommendations in research could be implementation-oriented or further research-oriented.   

4. Your research: Research recommendations can be based on your topic, research objectives, literature review, and analysis, or evidence collected. For example, if your data points to the role of faculty involvement in developing effective programs, recommendations in research can include developing policies to increase faculty participation. Take a look at the evidence-based recommendation in research example s provided below.   

Table: Example of evidence-based research recommendation  

Avoid making the following mistakes when writing research recommendations :  

• Don’t undermine your own work: Recommendations in research should offer suggestions on how future studies can be built upon the current study as a natural extension of your work and not as an entirely new field of research.  
• Support your study arguments: Ensure that your research findings stand alone on their own merits to showcase the strength of your research paper.   

How to write recommendations in research

When writing research recommendations , your focus should be on highlighting what additional work can be done in that field. It gives direction to researchers, industries, or governments about changes or developments possible in this field. For example, recommendations in research can include practical and obtainable strategies offering suggestions to academia to address problems. It can also be a framework that helps government agencies in developing strategic or long-term plans for timely actions against disasters or aid nation-building.  

There are a few SMART 2 things to remember when writing recommendations in research. Your recommendations must be: 

• S pecific: Clearly state how challenges can be addressed for better outcomes and include an action plan that shows what can be achieved. 
• M easurable: Use verbs denoting measurable outcomes, such as identify, analyze, design, compute, assess, evaluate, revise, plan, etc., to strengthen recommendations in research .   
• A ttainable: Recommendations should offer a solution-oriented approach to problem-solving and must be written in a way that is easy to follow.  
• R elevant: Research recommendations should be reasonable, realistic, and result-based. Make sure to suggest future possibilities for your research field.  
• T imely: Time-based or time-sensitive recommendations in research help divide the action plan into long-term or short-term (immediate) goals. A timeline can also inform potential readers of what developments should occur over time.  

If you are wondering how many words to include in your research recommendation , a general rule of thumb would be to set aside 5% of the total word count for writing research recommendations . Finally, when writing the research implications and recommendations , stick to the facts and avoid overstating or over-generalizing the study findings. Both should be supported by evidence gathered through your data analysis.  

References:  

• Schmidt, F. L., & Hunter, J. E. (1998). The validity and utility of selection methods in personnel psychology: Practical and theoretical implications of 85 years of research findings.  Psychological bulletin ,  124 (2), 262.
• Doran, G. T. (1981). There’s a S.M.A.R.T. way to write management’s goals and objectives.  Manag Rev ,  70 (11), 35-36.

Researcher.Life is a subscription-based platform that unifies the best AI tools and services designed to speed up, simplify, and streamline every step of a researcher’s journey. The Researcher.Life All Access Pack is a one-of-a-kind subscription that unlocks full access to an AI writing assistant, literature recommender, journal finder, scientific illustration tool, and exclusive discounts on professional publication services from Editage.  

Based on 21+ years of experience in academia, Researcher.Life All Access empowers researchers to put their best research forward and move closer to success. Explore our top AI Tools pack, AI Tools + Publication Services pack, or Build Your Own Plan. Find everything a researcher needs to succeed, all in one place –  Get All Access now starting at just $17 a month !    

Related Posts

Independent vs Dependent Variables: Definitions & Examples

15 Benefits for Researchers with One All Access Subscription 

• - Google Chrome

Intended for healthcare professionals

• Access provided by Google Indexer
• My email alerts
• BMA member login
• Username * Password * Forgot your log in details? Need to activate BMA Member Log In Log in via OpenAthens Log in via your institution

Search form

• Advanced search
• Search responses
• Search blogs
• How to formulate...

How to formulate research recommendations

• Related content
• Peer review
• Polly Brown ( pbrown{at}bmjgroup.com ) , publishing manager 1 ,
• Klara Brunnhuber , clinical editor 1 ,
• Kalipso Chalkidou , associate director, research and development 2 ,
• Iain Chalmers , director 3 ,
• Mike Clarke , director 4 ,
• Mark Fenton , editor 3 ,
• Carol Forbes , reviews manager 5 ,
• Julie Glanville , associate director/information service manager 5 ,
• Nicholas J Hicks , consultant in public health medicine 6 ,
• Janet Moody , identification and prioritisation manager 6 ,
• Sara Twaddle , director 7 ,
• Hazim Timimi , systems developer 8 ,
• Pamela Young , senior programme manager 6
• 1 BMJ Publishing Group, London WC1H 9JR,
• 2 National Institute for Health and Clinical Excellence, London WC1V 6NA,
• 3 Database of Uncertainties about the Effects of Treatments, James Lind Alliance Secretariat, James Lind Initiative, Oxford OX2 7LG,
• 4 UK Cochrane Centre, Oxford OX2 7LG,
• 5 Centre for Reviews and Dissemination, University of York, York YO10 5DD,
• 6 National Coordinating Centre for Health Technology Assessment, University of Southampton, Southampton SO16 7PX,
• 7 Scottish Intercollegiate Guidelines Network, Edinburgh EH2 1EN,
• 8 Update Software, Oxford OX2 7LG
• Correspondence to: PBrown
• Accepted 22 September 2006

“More research is needed” is a conclusion that fits most systematic reviews. But authors need to be more specific about what exactly is required

Long awaited reports of new research, systematic reviews, and clinical guidelines are too often a disappointing anticlimax for those wishing to use them to direct future research. After many months or years of effort and intellectual energy put into these projects, authors miss the opportunity to identify unanswered questions and outstanding gaps in the evidence. Most reports contain only a less than helpful, general research recommendation. This means that the potential value of these recommendations is lost.

Current recommendations

In 2005, representatives of organisations commissioning and summarising research, including the BMJ Publishing Group, the Centre for Reviews and Dissemination, the National Coordinating Centre for Health Technology Assessment, the National Institute for Health and Clinical Excellence, the Scottish Intercollegiate Guidelines Network, and the UK Cochrane Centre, met as members of the development group for the Database of Uncertainties about the Effects of Treatments (see bmj.com for details on all participating organisations). Our aim was to discuss the state of research recommendations within our organisations and to develop guidelines for improving the presentation of proposals for further research. All organisations had found weaknesses in the way researchers and authors of systematic reviews and clinical guidelines stated the need for further research. As part of the project, a member of the Centre for Reviews and Dissemination under-took a rapid literature search to identify information on research recommendation models, which found some individual methods but no group initiatives to attempt to standardise recommendations.

Suggested format for research recommendations on the effects of treatments

Core elements.

E Evidence (What is the current state of the evidence?)

P Population (What is the population of interest?)

I Intervention (What are the interventions of interest?)

C Comparison (What are the comparisons of interest?)

O Outcome (What are the outcomes of interest?)

T Time stamp (Date of recommendation)

Optional elements

d Disease burden or relevance

t Time aspect of core elements of EPICOT

s Appropriate study type according to local need

In January 2006, the National Coordinating Centre for Health Technology Assessment presented the findings of an initial comparative analysis of how different organisations currently structure their research recommendations. The National Institute for Health and Clinical Excellence and the National Coordinating Centre for Health Technology Assessment request authors to present recommendations in a four component format for formulating well built clinical questions around treatments: population, intervention, comparison, and outcomes (PICO). 1 In addition, the research recommendation is dated and authors are asked to provide the current state of the evidence to support the proposal.

Clinical Evidence , although not directly standardising its sections for research recommendations, presents gaps in the evidence using a slightly extended version of the PICO format: evidence, population, intervention, comparison, outcomes, and time (EPICOT). Clinical Evidence has used this inherent structure to feed research recommendations on interventions categorised as “unknown effectiveness” back to the National Coordinating Centre for Health Technology Assessment and for inclusion in the Database of Uncertainties about the Effects of Treatments ( http://www.duets.nhs.uk/ ).

We decided to propose the EPICOT format as the basis for its statement on formulating research recommendations and tested this proposal through discussion and example. We agreed that this set of components provided enough context for formulating research recommendations without limiting researchers. In order for the proposed framework to be flexible and more widely applicable, the group discussed using several optional components when they seemed relevant or were proposed by one or more of the group members. The final outcome of discussions resulted in the proposed EPICOT+ format (box).

A recent BMJ article highlighted how lack of research hinders the applicability of existing guidelines to patients in primary care who have had a stroke or transient ischaemic attack. 2 Most research in the area had been conducted in younger patients with a recent episode and in a hospital setting. The authors concluded that “further evidence should be collected on the efficacy and adverse effects of intensive blood pressure lowering in representative populations before we implement this guidance [from national and international guidelines] in primary care.” Table 1 outlines how their recommendations could be formulated using the EPICOT+ format. The decision on whether additional research is indeed clinically and ethically warranted will still lie with the organisation considering commissioning the research.

Research recommendation based on gap in the evidence identified by a cross sectional study of clinical guidelines for management of patients who have had a stroke

• View inline

Table 2 shows the use of EPICOT+ for an unanswered question on the effectiveness of compliance therapy in people with schizophrenia, identified by the Database of Uncertainties about the Effects of Treatments.

Research recommendation based on a gap in the evidence on treatment of schizophrenia identified by the Database of Uncertainties about the Effects of Treatments

Discussions around optional elements

Although the group agreed that the PICO elements should be core requirements for a research recommendation, intense discussion centred on the inclusion of factors defining a more detailed context, such as current state of evidence (E), appropriate study type (s), disease burden and relevance (d), and timeliness (t).

Initially, group members interpreted E differently. Some viewed it as the supporting evidence for a research recommendation and others as the suggested study type for a research recommendation. After discussion, we agreed that E should be used to refer to the amount and quality of research supporting the recommendation. However, the issue remained contentious as some of us thought that if a systematic review was available, its reference would sufficiently identify the strength of the existing evidence. Others thought that adding evidence to the set of core elements was important as it provided a summary of the supporting evidence, particularly as the recommendation was likely to be abstracted and used separately from the review or research that led to its formulation. In contrast, the suggested study type (s) was left as an optional element.

A research recommendation will rarely have an absolute value in itself. Its relative priority will be influenced by the burden of ill health (d), which is itself dependent on factors such as local prevalence, disease severity, relevant risk factors, and the priorities of the organisation considering commissioning the research.

Similarly, the issue of time (t) could be seen to be relevant to each of the core elements in varying ways—for example, duration of treatment, length of follow-up. The group therefore agreed that time had a subsidiary role within each core item; however, T as the date of the recommendation served to define its shelf life and therefore retained individual importance.

Applicability and usability

The proposed statement on research recommendations applies to uncertainties of the effects of any form of health intervention or treatment and is intended for research in humans rather than basic scientific research. Further investigation is required to assess the applicability of the format for questions around diagnosis, signs and symptoms, prognosis, investigations, and patient preference.

When the proposed format is applied to a specific research recommendation, the emphasis placed on the relevant part(s) of the EPICOT+ format may vary by author, audience, and intended purpose. For example, a recommendation for research into treatments for transient ischaemic attack may or may not define valid outcome measures to assess quality of life or gather data on adverse effects. Among many other factors, its implementation will also depend on the strength of current findings—that is, strong evidence may support a tightly focused recommendation whereas a lack of evidence would result in a more general recommendation.

The controversy within the group, especially around the optional components, reflects the different perspectives of the participating organisations—whether they were involved in commissioning, undertaking, or summarising research. Further issues will arise during the implementation of the proposed format, and we welcome feedback and discussion.

Summary points

No common guidelines exist for the formulation of recommendations for research on the effects of treatments

Major organisations involved in commissioning or summarising research compared their approaches and agreed on core questions

The essential items can be summarised as EPICOT+ (evidence, population, intervention, comparison, outcome, and time)

Further details, such as disease burden and appropriate study type, should be considered as required

We thank Patricia Atkinson and Jeremy Wyatt.

Contributors and sources All authors contributed to manuscript preparation and approved the final draft. NJH is the guarantor.

Competing interests None declared.

• Richardson WS ,
• Wilson MC ,
• Nishikawa J ,
• Hayward RSA
• McManus RJ ,
• Leonardi-Bee J ,
• PROGRESS Collaborative Group
• Warburton E
• Rothwell P ,
• McIntosh AM ,
• Lawrie SM ,
• Stanfield AC
• O'Donnell C ,
• Donohoe G ,
• Sharkey L ,
• Jablensky A ,
• Sartorius N ,
• Ernberg G ,

Research Recommendations – Guiding policy-makers for evidence-based decision making

Research recommendations play a crucial role in guiding scholars and researchers toward fruitful avenues of exploration. In an era marked by rapid technological advancements and an ever-expanding knowledge base, refining the process of generating research recommendations becomes imperative.

But, what is a research recommendation?

Research recommendations are suggestions or advice provided to researchers to guide their study on a specific topic . They are typically given by experts in the field. Research recommendations are more action-oriented and provide specific guidance for decision-makers, unlike implications that are broader and focus on the broader significance and consequences of the research findings. However, both are crucial components of a research study.

Difference Between Research Recommendations and Implication

Although research recommendations and implications are distinct components of a research study, they are closely related. The differences between them are as follows:

Types of Research Recommendations

Recommendations in research can take various forms, which are as follows:

These recommendations aim to assist researchers in navigating the vast landscape of academic knowledge.

Let us dive deeper to know about its key components and the steps to write an impactful research recommendation.

Key Components of Research Recommendations

The key components of research recommendations include defining the research question or objective, specifying research methods, outlining data collection and analysis processes, presenting results and conclusions, addressing limitations, and suggesting areas for future research. Here are some characteristics of research recommendations:

Research recommendations offer various advantages and play a crucial role in ensuring that research findings contribute to positive outcomes in various fields. However, they also have few limitations which highlights the significance of a well-crafted research recommendation in offering the promised advantages.

The importance of research recommendations ranges in various fields, influencing policy-making, program development, product development, marketing strategies, medical practice, and scientific research. Their purpose is to transfer knowledge from researchers to practitioners, policymakers, or stakeholders, facilitating informed decision-making and improving outcomes in different domains.

How to Write Research Recommendations?

Research recommendations can be generated through various means, including algorithmic approaches, expert opinions, or collaborative filtering techniques. Here is a step-wise guide to build your understanding on the development of research recommendations.

1. Understand the Research Question:

Understand the research question and objectives before writing recommendations. Also, ensure that your recommendations are relevant and directly address the goals of the study.

2. Review Existing Literature:

Familiarize yourself with relevant existing literature to help you identify gaps , and offer informed recommendations that contribute to the existing body of research.

3. Consider Research Methods:

Evaluate the appropriateness of different research methods in addressing the research question. Also, consider the nature of the data, the study design, and the specific objectives.

4. Identify Data Collection Techniques:

Gather dataset from diverse authentic sources. Include information such as keywords, abstracts, authors, publication dates, and citation metrics to provide a rich foundation for analysis.

5. Propose Data Analysis Methods:

Suggest appropriate data analysis methods based on the type of data collected. Consider whether statistical analysis, qualitative analysis, or a mixed-methods approach is most suitable.

6. Consider Limitations and Ethical Considerations:

Acknowledge any limitations and potential ethical considerations of the study. Furthermore, address these limitations or mitigate ethical concerns to ensure responsible research.

7. Justify Recommendations:

Explain how your recommendation contributes to addressing the research question or objective. Provide a strong rationale to help researchers understand the importance of following your suggestions.

8. Summarize Recommendations:

Provide a concise summary at the end of the report to emphasize how following these recommendations will contribute to the overall success of the research project.

By following these steps, you can create research recommendations that are actionable and contribute meaningfully to the success of the research project.

Download now to unlock some tips to improve your journey of writing research recommendations.

Example of a Research Recommendation

Here is an example of a research recommendation based on a hypothetical research to improve your understanding.

Research Recommendation: Enhancing Student Learning through Integrated Learning Platforms

Background:

The research study investigated the impact of an integrated learning platform on student learning outcomes in high school mathematics classes. The findings revealed a statistically significant improvement in student performance and engagement when compared to traditional teaching methods.

Recommendation:

In light of the research findings, it is recommended that educational institutions consider adopting and integrating the identified learning platform into their mathematics curriculum. The following specific recommendations are provided:

• Implementation of the Integrated Learning Platform:

Schools are encouraged to adopt the integrated learning platform in mathematics classrooms, ensuring proper training for teachers on its effective utilization.

• Professional Development for Educators:

Develop and implement professional programs to train educators in the effective use of the integrated learning platform to address any challenges teachers may face during the transition.

• Monitoring and Evaluation:

Establish a monitoring and evaluation system to track the impact of the integrated learning platform on student performance over time.

• Resource Allocation:

Allocate sufficient resources, both financial and technical, to support the widespread implementation of the integrated learning platform.

By implementing these recommendations, educational institutions can harness the potential of the integrated learning platform and enhance student learning experiences and academic achievements in mathematics.

This example covers the components of a research recommendation, providing specific actions based on the research findings, identifying the target audience, and outlining practical steps for implementation.

Using AI in Research Recommendation Writing

Enhancing research recommendations is an ongoing endeavor that requires the integration of cutting-edge technologies, collaborative efforts, and ethical considerations. By embracing data-driven approaches and leveraging advanced technologies, the research community can create more effective and personalized recommendation systems. However, it is accompanied by several limitations. Therefore, it is essential to approach the use of AI in research with a critical mindset, and complement its capabilities with human expertise and judgment.

Here are some limitations of integrating AI in writing research recommendation and some ways on how to counter them.

1. Data Bias

AI systems rely heavily on data for training. If the training data is biased or incomplete, the AI model may produce biased results or recommendations.

How to tackle: Audit regularly the model’s performance to identify any discrepancies and adjust the training data and algorithms accordingly.

2. Lack of Understanding of Context:

AI models may struggle to understand the nuanced context of a particular research problem. They may misinterpret information, leading to inaccurate recommendations.

How to tackle: Use AI to characterize research articles and topics. Employ them to extract features like keywords, authorship patterns and content-based details.

3. Ethical Considerations:

AI models might stereotype certain concepts or generate recommendations that could have negative consequences for certain individuals or groups.

How to tackle: Incorporate user feedback mechanisms to reduce redundancies. Establish an ethics review process for AI models in research recommendation writing.

4. Lack of Creativity and Intuition:

AI may struggle with tasks that require a deep understanding of the underlying principles or the ability to think outside the box.

How to tackle: Hybrid approaches can be employed by integrating AI in data analysis and identifying patterns for accelerating the data interpretation process.

5. Interpretability:

Many AI models, especially complex deep learning models, lack transparency on how the model arrived at a particular recommendation.

How to tackle: Implement models like decision trees or linear models. Provide clear explanation of the model architecture, training process, and decision-making criteria.

6. Dynamic Nature of Research:

Research fields are dynamic, and new information is constantly emerging. AI models may struggle to keep up with the rapidly changing landscape and may not be able to adapt to new developments.

How to tackle: Establish a feedback loop for continuous improvement. Regularly update the recommendation system based on user feedback and emerging research trends.

The integration of AI in research recommendation writing holds great promise for advancing knowledge and streamlining the research process. However, navigating these concerns is pivotal in ensuring the responsible deployment of these technologies. Researchers need to understand the use of responsible use of AI in research and must be aware of the ethical considerations.

Exploring research recommendations plays a critical role in shaping the trajectory of scientific inquiry. It serves as a compass, guiding researchers toward more robust methodologies, collaborative endeavors, and innovative approaches. Embracing these suggestions not only enhances the quality of individual studies but also contributes to the collective advancement of human understanding.

Frequently Asked Questions

The purpose of recommendations in research is to provide practical and actionable suggestions based on the study's findings, guiding future actions, policies, or interventions in a specific field or context. Recommendations bridges the gap between research outcomes and their real-world application.

To make a research recommendation, analyze your findings, identify key insights, and propose specific, evidence-based actions. Include the relevance of the recommendations to the study's objectives and provide practical steps for implementation.

Begin a recommendation by succinctly summarizing the key findings of the research. Clearly state the purpose of the recommendation and its intended impact. Use a direct and actionable language to convey the suggested course of action.

Rate this article Cancel Reply

Your email address will not be published.

Enago Academy's Most Popular Articles

• Publishing Research
• Reporting Research

How to Optimize Your Research Process: A step-by-step guide

For researchers across disciplines, the path to uncovering novel findings and insights is often filled…

• Industry News
• Trending Now

Breaking Barriers: Sony and Nature unveil “Women in Technology Award”

Sony Group Corporation and the prestigious scientific journal Nature have collaborated to launch the inaugural…

Achieving Research Excellence: Checklist for good research practices

Academia is built on the foundation of trustworthy and high-quality research, supported by the pillars…

Digital Citations: A comprehensive guide to citing of websites in APA, MLA, and CMOS style

In today’s digital age, the internet serves as an invaluable resource for researchers across all…

• Promoting Research

Plain Language Summary — Communicating your research to bridge the academic-lay gap

Science can be complex, but does that mean it should not be accessible to the…

Digital Citations: A comprehensive guide to citing of websites in APA, MLA, and CMOS…

Choosing the Right Analytical Approach: Thematic analysis vs. content analysis for…

Beyond the Podium: Understanding the differences in conference and academic…

Sign-up to read more

Subscribe for free to get unrestricted access to all our resources on research writing and academic publishing including:

• 2000+ blog articles
• 50+ Webinars
• 10+ Expert podcasts
• 50+ Infographics
• 10+ Checklists
• Research Guides

We hate spam too. We promise to protect your privacy and never spam you.

I am looking for Editing/ Proofreading services for my manuscript Tentative date of next journal submission:

What should universities' stance be on AI tools in research and academic writing?

How To Write The Conclusion Chapter

The what, why & how explained simply (with examples).

By: Jenna Crossley (PhD Cand). Reviewed By: Dr. Eunice Rautenbach | September 2021

So, you’ve wrapped up your results and discussion chapters, and you’re finally on the home stretch – the conclusion chapter . In this post, we’ll discuss everything you need to know to craft a high-quality conclusion chapter for your dissertation or thesis project.

Overview: Dissertation Conclusion Chapter

• What the thesis/dissertation conclusion chapter is
• What to include in your conclusion chapter
• How to structure and write up your conclusion chapter
• A few tips  to help you ace the chapter

What exactly is the conclusion chapter?

The conclusion chapter is typically the final major chapter of a dissertation or thesis. As such, it serves as a concluding summary of your research findings and wraps up the document. While some publications such as journal articles and research reports combine the discussion and conclusion sections, these are typically separate chapters in a dissertation or thesis. As always, be sure to check what your university’s structural preference is before you start writing up these chapters.

So, what’s the difference between the discussion and the conclusion chapter?

Well, the two chapters are quite similar , as they both discuss the key findings of the study. However, the conclusion chapter is typically more general and high-level in nature. In your discussion chapter, you’ll typically discuss the intricate details of your study, but in your conclusion chapter, you’ll take a   broader perspective, reporting on the main research outcomes and how these addressed your research aim (or aims) .

A core function of the conclusion chapter is to synthesise all major points covered in your study and to tell the reader what they should take away from your work. Basically, you need to tell them what you found , why it’s valuable , how it can be applied , and what further research can be done.

Whatever you do, don’t just copy and paste what you’ve written in your discussion chapter! The conclusion chapter should not be a simple rehash of the discussion chapter. While the two chapters are similar, they have distinctly different functions.  

What should I include in the conclusion chapter?

To understand what needs to go into your conclusion chapter, it’s useful to understand what the chapter needs to achieve. In general, a good dissertation conclusion chapter should achieve the following:

• Summarise the key findings of the study
• Explicitly answer the research question(s) and address the research aims
• Inform the reader of the study’s main contributions
• Discuss any limitations or weaknesses of the study
• Present recommendations for future research

Therefore, your conclusion chapter needs to cover these core components. Importantly, you need to be careful not to include any new findings or data points. Your conclusion chapter should be based purely on data and analysis findings that you’ve already presented in the earlier chapters. If there’s a new point you want to introduce, you’ll need to go back to your results and discussion chapters to weave the foundation in there.

In many cases, readers will jump from the introduction chapter directly to the conclusions chapter to get a quick overview of the study’s purpose and key findings. Therefore, when you write up your conclusion chapter, it’s useful to assume that the reader hasn’t consumed the inner chapters of your dissertation or thesis. In other words, craft your conclusion chapter such that there’s a strong connection and smooth flow between the introduction and conclusion chapters, even though they’re on opposite ends of your document.

Need a helping hand?

How to write the conclusion chapter

Now that you have a clearer view of what the conclusion chapter is about, let’s break down the structure of this chapter so that you can get writing. Keep in mind that this is merely a typical structure – it’s not set in stone or universal. Some universities will prefer that you cover some of these points in the discussion chapter , or that you cover the points at different levels in different chapters.

Step 1: Craft a brief introduction section

As with all chapters in your dissertation or thesis, the conclusions chapter needs to start with a brief introduction. In this introductory section, you’ll want to tell the reader what they can expect to find in the chapter, and in what order . Here’s an example of what this might look like:

This chapter will conclude the study by summarising the key research findings in relation to the research aims and questions and discussing the value and contribution thereof. It will also review the limitations of the study and propose opportunities for future research.

Importantly, the objective here is just to give the reader a taste of what’s to come (a roadmap of sorts), not a summary of the chapter. So, keep it short and sweet – a paragraph or two should be ample.

Step 2: Discuss the overall findings in relation to the research aims

The next step in writing your conclusions chapter is to discuss the overall findings of your study , as they relate to the research aims and research questions . You would have likely covered similar ground in the discussion chapter, so it’s important to zoom out a little bit here and focus on the broader findings – specifically, how these help address the research aims .

In practical terms, it’s useful to start this section by reminding your reader of your research aims and research questions, so that the findings are well contextualised. In this section, phrases such as, “This study aimed to…” and “the results indicate that…” will likely come in handy. For example, you could say something like the following:

This study aimed to investigate the feeding habits of the naked mole-rat. The results indicate that naked mole rats feed on underground roots and tubers. Further findings show that these creatures eat only a part of the plant, leaving essential parts to ensure long-term food stability.

Be careful not to make overly bold claims here. Avoid claims such as “this study proves that” or “the findings disprove existing the existing theory”. It’s seldom the case that a single study can prove or disprove something. Typically, this is achieved by a broader body of research, not a single study – especially not a dissertation or thesis which will inherently have significant and limitations. We’ll discuss those limitations a little later.

Step 3: Discuss how your study contributes to the field

Next, you’ll need to discuss how your research has contributed to the field – both in terms of theory and practice . This involves talking about what you achieved in your study, highlighting why this is important and valuable, and how it can be used or applied.

In this section you’ll want to:

• Mention any research outputs created as a result of your study (e.g., articles, publications, etc.)
• Inform the reader on just how your research solves your research problem , and why that matters
• Reflect on gaps in the existing research and discuss how your study contributes towards addressing these gaps
• Discuss your study in relation to relevant theories . For example, does it confirm these theories or constructively challenge them?
• Discuss how your research findings can be applied in the real world . For example, what specific actions can practitioners take, based on your findings?

Be careful to strike a careful balance between being firm but humble in your arguments here. It’s unlikely that your one study will fundamentally change paradigms or shake up the discipline, so making claims to this effect will be frowned upon . At the same time though, you need to present your arguments with confidence, firmly asserting the contribution your research has made, however small that contribution may be. Simply put, you need to keep it balanced .

Step 4: Reflect on the limitations of your study

Now that you’ve pumped your research up, the next step is to critically reflect on the limitations and potential shortcomings of your study. You may have already covered this in the discussion chapter, depending on your university’s structural preferences, so be careful not to repeat yourself unnecessarily.

There are many potential limitations that can apply to any given study. Some common ones include:

• Sampling issues that reduce the generalisability of the findings (e.g., non-probability sampling )
• Insufficient sample size (e.g., not getting enough survey responses ) or limited data access
• Low-resolution data collection or analysis techniques
• Researcher bias or lack of experience
• Lack of access to research equipment
• Time constraints that limit the methodology (e.g. cross-sectional vs longitudinal time horizon)
• Budget constraints that limit various aspects of the study

Discussing the limitations of your research may feel self-defeating (no one wants to highlight their weaknesses, right), but it’s a critical component of high-quality research. It’s important to appreciate that all studies have limitations (even well-funded studies by expert researchers) – therefore acknowledging these limitations adds credibility to your research by showing that you understand the limitations of your research design .

That being said, keep an eye on your wording and make sure that you don’t undermine your research . It’s important to strike a balance between recognising the limitations, but also highlighting the value of your research despite those limitations. Show the reader that you understand the limitations, that these were justified given your constraints, and that you know how they can be improved upon – this will get you marks.

Next, you’ll need to make recommendations for future studies. This will largely be built on the limitations you just discussed. For example, if one of your study’s weaknesses was related to a specific data collection or analysis method, you can make a recommendation that future researchers undertake similar research using a more sophisticated method.

Another potential source of future research recommendations is any data points or analysis findings that were interesting or surprising , but not directly related to your study’s research aims and research questions. So, if you observed anything that “stood out” in your analysis, but you didn’t explore it in your discussion (due to a lack of relevance to your research aims), you can earmark that for further exploration in this section.

Essentially, this section is an opportunity to outline how other researchers can build on your study to take the research further and help develop the body of knowledge. So, think carefully about the new questions that your study has raised, and clearly outline these for future researchers to pick up on.

Step 6: Wrap up with a closing summary

Quick tips for a top-notch conclusion chapter

Now that we’ve covered the what , why and how of the conclusion chapter, here are some quick tips and suggestions to help you craft a rock-solid conclusion.

• Don’t ramble . The conclusion chapter usually consumes 5-7% of the total word count (although this will vary between universities), so you need to be concise. Edit this chapter thoroughly with a focus on brevity and clarity.
• Be very careful about the claims you make in terms of your study’s contribution. Nothing will make the marker’s eyes roll back faster than exaggerated or unfounded claims. Be humble but firm in your claim-making.
• Use clear and simple language that can be easily understood by an intelligent layman. Remember that not every reader will be an expert in your field, so it’s important to make your writing accessible. Bear in mind that no one knows your research better than you do, so it’s important to spell things out clearly for readers.

Hopefully, this post has given you some direction and confidence to take on the conclusion chapter of your dissertation or thesis with confidence. If you’re still feeling a little shaky and need a helping hand, consider booking a free initial consultation with a friendly Grad Coach to discuss how we can help you with hands-on, private coaching.

Psst… there’s more (for free)

This post is part of our dissertation mini-course, which covers everything you need to get started with your dissertation, thesis or research project. 

You Might Also Like:

17 Comments

Really you team are doing great!

Your guide on writing the concluding chapter of a research is really informative especially to the beginners who really do not know where to start. Im now ready to start. Keep it up guys

Really your team are doing great!

Very helpful guidelines, timely saved. Thanks so much for the tips.

This post was very helpful and informative. Thank you team.

A very enjoyable, understandable and crisp presentation on how to write a conclusion chapter. I thoroughly enjoyed it. Thanks Jenna.

This was a very helpful article which really gave me practical pointers for my concluding chapter. Keep doing what you are doing! It meant a lot to me to be able to have this guide. Thank you so much.

Nice content dealing with the conclusion chapter, it’s a relief after the streneous task of completing discussion part.Thanks for valuable guidance

Thanks for your guidance

I get all my doubts clarified regarding the conclusion chapter. It’s really amazing. Many thanks.

Very helpful tips. Thanks so much for the guidance

Thank you very much for this piece. It offers a very helpful starting point in writing the conclusion chapter of my thesis.

It’s awesome! Most useful and timely too. Thanks a million times

Bundle of thanks for your guidance. It was greatly helpful.

Wonderful, clear, practical guidance. So grateful to read this as I conclude my research. Thank you.

Submit a Comment Cancel reply

Your email address will not be published. Required fields are marked *

Save my name, email, and website in this browser for the next time I comment.

• Print Friendly

• Cookies & Privacy
• GETTING STARTED
• Introduction
• FUNDAMENTALS
• Acknowledgements
• Research questions & hypotheses
• Concepts, constructs & variables
• Research limitations
• Getting started
• Sampling Strategy
• Research Quality
• Research Ethics
• Data Analysis

FUTURE RESEARCH

Types of future research suggestion.

The Future Research section of your dissertation is often combined with the Research Limitations section of your final, Conclusions chapter. This is because your future research suggestions generally arise out of the research limitations you have identified in your own dissertation. In this article, we discuss six types of future research suggestion. These include: (1) building on a particular finding in your research; (2) addressing a flaw in your research; examining (or testing) a theory (framework or model) either (3) for the first time or (4) in a new context, location and/or culture; (5) re-evaluating and (6) expanding a theory (framework or model). The goal of the article is to help you think about the potential types of future research suggestion that you may want to include in your dissertation.

Before we discuss each of these types of future research suggestion, we should explain why we use the word examining and then put or testing in brackets. This is simply because the word examining may be considered more appropriate when students use a qualitative research design; whereas the word testing fits better with dissertations drawing on a quantitative research design. We also put the words framework or model in brackets after the word theory . We do this because a theory , framework and model are not the same things. In the sections that follow, we discuss six types of future research suggestion.

Addressing research limitations in your dissertation

Building on a particular finding or aspect of your research, examining a conceptual framework (or testing a theoretical model) for the first time, examining a conceptual framework (or testing a theoretical model) in a new context, location and/or culture.

• Expanding a conceptual framework (or testing a theoretical model)

Re-evaluating a conceptual framework (or theoretical model)

In the Research Limitations section of your Conclusions chapter, you will have inevitably detailed the potential flaws (i.e., research limitations) of your dissertation. These may include:

An inability to answer your research questions

Theoretical and conceptual problems

Limitations of your research strategy

Problems of research quality

Identifying what these research limitations were and proposing future research suggestions that address them is arguably the easiest and quickest ways to complete the Future Research section of your Conclusions chapter.

Often, the findings from your dissertation research will highlight a number of new avenues that could be explored in future studies. These can be grouped into two categories:

Your dissertation will inevitably lead to findings that you did not anticipate from the start. These are useful when making future research suggestions because they can lead to entirely new avenues to explore in future studies. If this was the case, it is worth (a) briefly describing what these unanticipated findings were and (b) suggesting a research strategy that could be used to explore such findings in future.

Sometimes, dissertations manage to address all aspects of the research questions that were set. However, this is seldom the case. Typically, there will be aspects of your research questions that could not be answered. This is not necessarily a flaw in your research strategy, but may simply reflect that fact that the findings did not provide all the answers you hoped for. If this was the case, it is worth (a) briefly describing what aspects of your research questions were not answered and (b) suggesting a research strategy that could be used to explore such aspects in future.

You may want to recommend that future research examines the conceptual framework (or tests the theoretical model) that you developed. This is based on the assumption that the primary goal of your dissertation was to set out a conceptual framework (or build a theoretical model). It is also based on the assumption that whilst such a conceptual framework (or theoretical model) was presented, your dissertation did not attempt to examine (or test) it in the field . The focus of your dissertations was most likely a review of the literature rather than something that involved you conducting primary research.

Whilst it is quite rare for dissertations at the undergraduate and master's level to be primarily theoretical in nature like this, it is not unknown. If this was the case, you should think about how the conceptual framework (or theoretical model) that you have presented could be best examined (or tested) in the field . In understanding the how , you should think about two factors in particular:

What is the context, location and/or culture that would best lend itself to my conceptual framework (or theoretical model) if it were to be examined (or tested) in the field?

What research strategy is most appropriate to examine my conceptual framework (or test my theoretical model)?

If the future research suggestion that you want to make is based on examining your conceptual framework (or testing your theoretical model) in the field , you need to suggest the best scenario for doing so.

More often than not, you will not only have set out a conceptual framework (or theoretical model), as described in the previous section, but you will also have examined (or tested) it in the field . When you do this, focus is typically placed on a specific context, location and/or culture.

If this is the case, the obvious future research suggestion that you could propose would be to examine your conceptual framework (or test the theoretical model) in a new context, location and/or culture. For example, perhaps you focused on consumers (rather than businesses), or Canada (rather than the United Kingdom), or a more individualistic culture like the United States (rather than a more collectivist culture like China).

When you propose a new context, location and/or culture as your future research suggestion, make sure you justify the choice that you make. For example, there may be little value in future studies looking at different cultures if culture is not an important component underlying your conceptual framework (or theoretical model). If you are not sure whether a new context, location or culture is more appropriate, or what new context, location or culture you should select, a review the literature will often help clarify where you focus should be.

Expanding a conceptual framework (or theoretical model)

Assuming that you have set out a conceptual framework (or theoretical model) and examined (or tested) it in the field , another series of future research suggestions comes out of expanding that conceptual framework (or theoretical model).

We talk about a series of future research suggestions because there are so many ways that you can expand on your conceptual framework (or theoretical model). For example, you can do this by:

Examining constructs (or variables) that were included in your conceptual framework (or theoretical model) but were not focused.

Looking at a particular relationship aspect of your conceptual framework (or theoretical model) further.

Adding new constructs (or variables) to the conceptual framework (or theoretical model) you set out (if justified by the literature).

It would be possible to include one or a number of these as future research suggestions. Again, make sure that any suggestions you make have are justified , either by your findings or the literature.

With the dissertation process at the undergraduate and master's level lasting between 3 and 9 months, a lot a can happen in between. For example, a specific event (e.g., 9/11, the economic crisis) or some new theory or evidence that undermines (or questions) the literature (theory) and assumptions underpinning your conceptual framework (or theoretical model). Clearly, there is little you can do about this. However, if this happens, reflecting on it and re-evaluating your conceptual framework (or theoretical model), as well as your findings, is an obvious source of future research suggestions.

Implications or Recommendations in Research: What's the Difference?

• Peer Review

High-quality research articles that get many citations contain both implications and recommendations. Implications are the impact your research makes, whereas recommendations are specific actions that can then be taken based on your findings, such as for more research or for policymaking.

Updated on August 23, 2022

That seems clear enough, but the two are commonly confused.

This confusion is especially true if you come from a so-called high-context culture in which information is often implied based on the situation, as in many Asian cultures. High-context cultures are different from low-context cultures where information is more direct and explicit (as in North America and many European cultures).

Let's set these two straight in a low-context way; i.e., we'll be specific and direct! This is the best way to be in English academic writing because you're writing for the world.

Implications and recommendations in a research article

The standard format of STEM research articles is what's called IMRaD:

• Introduction
• Discussion/conclusions

Some journals call for a separate conclusions section, while others have the conclusions as the last part of the discussion. You'll write these four (or five) sections in the same sequence, though, no matter the journal.

The discussion section is typically where you restate your results and how well they confirmed your hypotheses. Give readers the answer to the questions for which they're looking to you for an answer.

At this point, many researchers assume their paper is finished. After all, aren't the results the most important part? As you might have guessed, no, you're not quite done yet.

The discussion/conclusions section is where to say what happened and what should now happen

The discussion/conclusions section of every good scientific article should contain the implications and recommendations.

The implications, first of all, are the impact your results have on your specific field. A high-impact, highly cited article will also broaden the scope here and provide implications to other fields. This is what makes research cross-disciplinary.

Recommendations, however, are suggestions to improve your field based on your results.

These two aspects help the reader understand your broader content: How and why your work is important to the world. They also tell the reader what can be changed in the future based on your results.

These aspects are what editors are looking for when selecting papers for peer review.

Implications and recommendations are, thus, written at the end of the discussion section, and before the concluding paragraph. They help to “wrap up” your paper. Once your reader understands what you found, the next logical step is what those results mean and what should come next.

Then they can take the baton, in the form of your work, and run with it. That gets you cited and extends your impact!

The order of implications and recommendations also matters. Both are written after you've summarized your main findings in the discussion section. Then, those results are interpreted based on ongoing work in the field. After this, the implications are stated, followed by the recommendations.

Writing an academic research paper is a bit like running a race. Finish strong, with your most important conclusion (recommendation) at the end. Leave readers with an understanding of your work's importance. Avoid generic, obvious phrases like "more research is needed to fully address this issue." Be specific.

The main differences between implications and recommendations (table)

Now let's dig a bit deeper into actually how to write these parts.

What are implications?

Research implications tell us how and why your results are important for the field at large. They help answer the question of “what does it mean?” Implications tell us how your work contributes to your field and what it adds to it. They're used when you want to tell your peers why your research is important for ongoing theory, practice, policymaking, and for future research.

Crucially, your implications must be evidence-based. This means they must be derived from the results in the paper.

Implications are written after you've summarized your main findings in the discussion section. They come before the recommendations and before the concluding paragraph. There is no specific section dedicated to implications. They must be integrated into your discussion so that the reader understands why the results are meaningful and what they add to the field.

A good strategy is to separate your implications into types. Implications can be social, political, technological, related to policies, or others, depending on your topic. The most frequently used types are theoretical and practical. Theoretical implications relate to how your findings connect to other theories or ideas in your field, while practical implications are related to what we can do with the results.

Key features of implications

• State the impact your research makes
• Helps us understand why your results are important
• Must be evidence-based
• Written in the discussion, before recommendations
• Can be theoretical, practical, or other (social, political, etc.)

Examples of implications

Let's take a look at some examples of research results below with their implications.

The result : one study found that learning items over time improves memory more than cramming material in a bunch of information at once .

The implications : This result suggests memory is better when studying is spread out over time, which could be due to memory consolidation processes.

The result : an intervention study found that mindfulness helps improve mental health if you have anxiety.

The implications : This result has implications for the role of executive functions on anxiety.

The result : a study found that musical learning helps language learning in children .

The implications : these findings suggest that language and music may work together to aid development.

What are recommendations?

As noted above, explaining how your results contribute to the real world is an important part of a successful article.

Likewise, stating how your findings can be used to improve something in future research is equally important. This brings us to the recommendations.

Research recommendations are suggestions and solutions you give for certain situations based on your results. Once the reader understands what your results mean with the implications, the next question they need to know is "what's next?"

Recommendations are calls to action on ways certain things in the field can be improved in the future based on your results. Recommendations are used when you want to convey that something different should be done based on what your analyses revealed.

Similar to implications, recommendations are also evidence-based. This means that your recommendations to the field must be drawn directly from your results.

The goal of the recommendations is to make clear, specific, and realistic suggestions to future researchers before they conduct a similar experiment. No matter what area your research is in, there will always be further research to do. Try to think about what would be helpful for other researchers to know before starting their work.

Recommendations are also written in the discussion section. They come after the implications and before the concluding paragraphs. Similar to the implications, there is usually no specific section dedicated to the recommendations. However, depending on how many solutions you want to suggest to the field, they may be written as a subsection.

Key features of recommendations

• Statements about what can be done differently in the field based on your findings
• Must be realistic and specific
• Written in the discussion, after implications and before conclusions
• Related to both your field and, preferably, a wider context to the research

Examples of recommendations

Here are some research results and their recommendations.

A meta-analysis found that actively recalling material from your memory is better than simply re-reading it .

• The recommendation: Based on these findings, teachers and other educators should encourage students to practice active recall strategies.

A medical intervention found that daily exercise helps prevent cardiovascular disease .

• The recommendation: Based on these results, physicians are recommended to encourage patients to exercise and walk regularly. Also recommended is to encourage more walking through public health offices in communities.

A study found that many research articles do not contain the sample sizes needed to statistically confirm their findings .

The recommendation: To improve the current state of the field, researchers should consider doing power analysis based on their experiment's design.

What else is important about implications and recommendations?

When writing recommendations and implications, be careful not to overstate the impact of your results. It can be tempting for researchers to inflate the importance of their findings and make grandiose statements about what their work means.

Remember that implications and recommendations must be coming directly from your results. Therefore, they must be straightforward, realistic, and plausible.

Another good thing to remember is to make sure the implications and recommendations are stated clearly and separately. Do not attach them to the endings of other paragraphs just to add them in. Use similar example phrases as those listed in the table when starting your sentences to clearly indicate when it's an implication and when it's a recommendation.

When your peers, or brand-new readers, read your paper, they shouldn't have to hunt through your discussion to find the implications and recommendations. They should be clear, visible, and understandable on their own.

That'll get you cited more, and you'll make a greater contribution to your area of science while extending the life and impact of your work.

The AJE Team

See our "Privacy Policy"

Suggestions for Future Research

Your dissertation needs to include suggestions for future research. Depending on requirements of your university, suggestions for future research can be either integrated into Research Limitations section or it can be a separate section.

You will need to propose 4-5 suggestions for future studies and these can include the following:

1. Building upon findings of your research . These may relate to findings of your study that you did not anticipate. Moreover, you may suggest future research to address unanswered aspects of your research problem.

2. Addressing limitations of your research . Your research will not be free from limitations and these may relate to formulation of research aim and objectives, application of data collection method, sample size, scope of discussions and analysis etc. You can propose future research suggestions that address the limitations of your study.

3. Constructing the same research in a new context, location and/or culture . It is most likely that you have addressed your research problem within the settings of specific context, location and/or culture. Accordingly, you can propose future studies that can address the same research problem in a different settings, context, location and/or culture.

4. Re-assessing and expanding theory, framework or model you have addressed in your research . Future studies can address the effects of specific event, emergence of a new theory or evidence and/or other recent phenomenon on your research problem.

My e-book,  The Ultimate Guide to Writing a Dissertation in Business Studies: a step by step assistance  offers practical assistance to complete a dissertation with minimum or no stress. The e-book covers all stages of writing a dissertation starting from the selection to the research area to submitting the completed version of the work within the deadline. John Dudovskiy

Improving the Nation's Water Security: Opportunities for Research (2007)

Chapter: 6 recommendations for future research directions, 6 recommendations for future research directions.

Progress has been made in the Environmental Protection Agency’s (EPA’s) water security research program (see Chapter 4 ), but many important research questions and technical support needs remain. In Chapter 3 , a framework is suggested for evaluating water security research initiatives that gives priority to research that improves response and recovery and/or develops risk reduction or consequence mitigation measures. The research should also produce tools with a reasonable likelihood of implementation and, where feasible, dual-use benefits. Based on this framework and the review of water security efforts already under way, two important water security research gaps are identified and discussed briefly in this chapter. In addition, short- and long-term water security research recommendations are made. The research recommendations are organized in this chapter according to the three long-term program objectives proposed in Chapter 5 emphasizing pre-incident, incident, and post-incident applications: (1) develop products to support more resilient design and operation of facilities and systems, (2) improve the ability of operators and responders to detect and assess incidents, and (3) improve response and recovery. Both drinking water and wastewater research priorities are addressed together within these three objectives to maximize research synergies that may exist.

KEY RESEARCH GAPS

The Water Security Research and Technical Support Action Plan (EPA, 2004a) set out a comprehensive guide for the EPA’s near-term research initiatives. Although the Action Plan was intended to provide a short-term (three- to four- year) research agenda, the previous National Research Council review (NRC, 2004) noted that several of the Action Plan projects represented long-term research questions not easily ad-

dressed in the original time frame. Therefore, the Action Plan provides a reasonable starting point for building the EPA’s future research program. Nevertheless, the short-term planning horizon of the Action Plan prevented consideration of two key subjects that are critical to a long-term water security research program: behavioral science and innovative system design. The committee recommends the EPA work in collaboration with other organizations to build research initiatives in these two areas.

Behavioral Science

The threat of bioterrorism presents new and different types of risks that are dynamic and pose difficult trade-offs, bringing about intellectual challenges and an emotional valence possibly more important than the risks themselves. Developing an effective communication strategy that meets the needs of the broad range of stakeholders (e.g., response organizations, water organizations and utilities, public health agencies, the public, the media) while addressing security concerns is clearly a high-priority research area. The EPA’s work on risk communication is focused primarily on the development of guidance, protocols, and training, and little emphasis has been devoted to interdisciplinary behavioral science research to better prepare stakeholders for water security incidents or to build confidence in their ability to respond. Behavioral science research could help address, for example, what the public’s beliefs, opinions, and knowledge about water security risks are; how risk perception and other psychological factors affect responses to water-related events; and how to communicate these risks with the public (Gray and Ropeik, 2002; Means, 2002; Roberson and Morely, 2005b). A better understanding of what short-term disruptions customers are prepared to tolerate may also guide response and recovery planning and the development of recovery technologies.

Previous experience with natural disasters and environmental risks provides a basis for investigating and predicting human behavior in risky situations (Fischoff, 2005). Existing models of human behavior during other kinds of crises, however, may not be adequate to forecast human behavior during bioterrorism or water security incidents (DiGiovanni et al., 2003).

Risk communicators consider empirical findings from psychology, cognitive science, communications, and other behavioral and social sciences to varying extents (Bostrom and Lofstedt, 2003). Although decision makers frequently predict panic and irrational behavior in times of

crisis, behavioral science researchers have found that people respond reasonably to such challenges (e.g., Fishoff, 2005). Given the urgency of terror risk communication, risk communicators are obliged to incorporate existing behavioral science research as it relates to water security risks.

The EPA should take advantage of existing behavioral science research that may be applicable to water security issues, but this requires knowledge and experience in behavioral science research. Where gaps exist, the EPA will need to engage in interdisciplinary, rigorous empirical research to obtain the necessary knowledge.

Innovative Designs for Secure and Resilient Water and Wastewater Systems

Innovative designs for water and wastewater infrastructure were not addressed in the EPA Action Plan, but the topic deserves a place in a long-term water security research program. The EPA’s research mission has traditionally included the development and testing of new concepts, technologies, and management structures for water and wastewater utilities to achieve practical objectives in public health, sustainability and cost-effectiveness. The addition of homeland security to its mission provides a unique opportunity to take a holistic view of current design and management of water and wastewater infrastructures. Innovation is needed to address the problem of aging infrastructures while making new water systems more resilient to natural hazards and malicious incidents. The EPA should, therefore, take a leadership role in providing guidance for the planning, design, and implementation of new, more sustainable and resilient water and wastewater facilities for the 21st century.

Disagreggation of large water and wastewater systems should be an overarching theme of innovation. Large and complex systems have developed in the United States following the pattern of urban and suburban sprawl. While there are clear economies of scale for large utilities in construction and system management, there are distinct disadvantages as well. The complexity of large systems makes security measures difficult to implement and complicates the response to an attack. For example, locating the source of incursion within the distribution system and isolating contaminated sections are more difficult in large and complex water systems. Long water residence times are also more likely to occur in large drinking water systems, and, as a result, disinfectant residual may be lacking in the extremities of the system because of the chemical and biological reactions that occur during transport. From a security perspec-

tive, inadequate disinfectant residual means less protection against intentional contamination by a microbial agent.

A breadth of possibilities exists for improving security through innovative infrastructure design. Satellite water treatment plants could boost water quality. Strategic placement of treatment devices (e.g., ultraviolet lamp arrays) within the distribution system could counter a bioterrorism attack. Wastewater treatment systems could be interconnected to provide more flexibility in case of attack, and diversion devices could be installed to isolate contaminants. Box 6-1 describes some of these concepts in greater detail, and specific research recommendations are suggested in the following section.

RESEARCH RECOMMENDATIONS: DEVELOP PRODUCTS TO SUPPORT MORE RESILIENT DESIGN AND OPERATION OF FACILITIES AND SYSTEMS

Specific research topics are suggested here in two areas to support development of more resilient water and wastewater systems: (1) innovative designs for water and wastewater and (2) improved methods for risk assessment, including processes for threat and consequence assessments.

Innovative Designs for Water and Wastewater Systems

Innovative changes to water infrastructure will require long-term investment in research. Existing systems have been in place for more than a century in older cities. Thus, bold new directions will understandably require intensive research at the outset to produce a defensible economic argument for change. On the other hand, the EPA has the opportunity to develop innovative approaches that can be implemented almost immediately in relatively new, as well as planned, urban and suburban areas. The first step in research would be to enumerate the opportunities for innovation, recognizing the constraints brought about by the size, age, and complexity of existing water and wastewater infrastructures. A broad-gauge, economic analysis should follow that would quantify the costs and multiple benefits of these innovative designs (e.g., increased security, improved drinking water quality, enhanced sustainability of water resources). In addition, there is an implicit need for EPA research-

ers to coordinate with the agency’s regulatory branch to validate the feasibility of the innovative concepts that are proposed.

Each of the infrastructure concepts illustrated in Box 6-1 require far more research to become feasible. The recommendations below outline specific research topics that, if addressed, could improve the safety and sustainability of water resources in the 21st century.

Disaggregation of Water and Wastewater Systems

The “distributed optimal technology network” (DOT-Net) concept (Norton and Weber, 2005; Weber, 2002; 2004) hinges upon the feasibility of distributed treatment via point-of-use (POU)/point-of-entry (POE) devices installed at the scale of individual buildings or perhaps small neighborhoods. The corollary premise is that installation of expensive advanced treatment technology at the centralized water treatment facility is unnecessary when only a fraction of the service area outside a “critical radius” requires additional protection. Only a broad economic analysis of this concept has been published thus far for a hypothetical urban center, but the assumptions need to be verified for actual systems, particularly because of the unique characteristics of individual cities. In addition, far more research is needed on the utility management required to ensure the reliability of POU/POE devices in widespread implementation.

Dual water systems have also been proposed to address aging infrastructure (see Box 6-1 ; Okun, 1997; 2005). As with the DOT-Net concept, long-term research is needed to determine the costs and benefits of constructing an entirely new paradigm for distribution system design. Research issues would include assessing the acceptability of reclaimed water for progressively more intense levels of nonpotable use (e.g., irrigation, toilet flushing, laundering), the acceptability and management demands of decentralized wastewater treatment facilities, and the net benefits to water security.

In-Pipe Interventions to Reduce Exposure

In-pipe engineering interventions (see Box 6-1 ) are deserving of research in a long-term water security research strategy. For example, research is needed to optimize the location of disinfection booster stations or to examine the effectiveness and feasibility of in situ ultraviolet (UV)

irradiation systems as a decontamination strategy. EPA research could also examine various pipe materials (e.g., stainless steel) and evaluate their benefits for security and sustainability relative to their costs.

Infrastructure Designs to Enable Isolation and Interconnection

Most large drinking water systems have the ability to isolate portions of their distribution systems during necessary system repairs, but security concerns provide a new impetus for rapid and effective isolation mechanisms. Research on innovative mechanisms to isolate or divert contaminated water in drinking water and wastewater systems would be useful. The EPA should identify these design options, research their costs and benefits (including dual-use benefits) and their feasibility both for existing systems and new infrastructure, and make this information available to system managers.

Improved Risk Assessments Procedures

A sound risk assessment process allows utilities to make better resource management decisions for enhancing their recovery capacity or security strategies to mitigate the consequences of an attack. The risk assessment process includes assessments of threat, consequences, and vulnerability. To date, most of the efforts to guide utilities in their own risk assessments have focused on vulnerabilities.

Threat Assessment

Water and wastewater utilities today are making resource management decisions related to security without adequate information about the nature and likelihood of threats to their systems. As discussed in Chapter 4 , the EPA has focused their efforts on identifying contaminant threats without conducting similarly detailed analyses of possible physical and cyber threats. Both the nature and likelihood of these threats are needed for efficient allocation of resources at the utility level and within the EPA’s research program. Improved threat assessment would require the EPA and/or a consortium of water experts to work closely with the intelligence community and local law enforcement agencies. Other national and federal laboratory expertise within the Department of Energy,

Department of Defense, and private-public community might be needed as well. Threat assessments for water and wastewater should be periodically reviewed to identify threat scenarios that should be added to the list and to remove those that are no longer a concern. The development of a threat assessment process for local water and wastewater utilities with current techniques used in other infrastructures would also be helpful, provided the threat information could be communicated to those who need it (ASME, 2004; Sandia National Laboratories, 2001).

Consequence Assessment

A consequence assessment should accompany the threat assessment within the risk assessment process. Consequence assessments would provide decision makers with information on the potential for fatalities, public health impacts, economic impacts, property damage, systems disruption, effects on other infrastructures, and loss of public confidence. Procedures for determining the expected consequences from an attack or natural disaster are not currently being systematically developed. As a result, water system managers do not have sufficient data to make decisions about the benefits of risk reduction relative to the costs. The development and application of a consequence assessment procedure would provide decision makers with information needed to decide whether to mitigate the consequences, upgrade with countermeasures, take steps to improve response and recovery capacity, and/or decide to accept the level of risk and take no further action. A fault tree analysis that includes, for example, options for redundant systems or contingency water supplies could provide vital information on whether to invest in security upgrades or less costly consequence mitigation strategies . Many of these approaches have already been developed for other infrastructures (e.g., Risk Assessment Methodology [RAM]-T for the high-voltage power transmission industry or RAM-D for dams, locks, and levees; see Sandia National Laboratories, 2001; 2002). A thorough review of other RAM methodologies could provide guidance for consequence assessment strategies that could be incorporated into the Risk Assessment Methodology for Water Utilities (RAM-W).

The EPA has worked to develop the AT Planner tool to assist utilities in assessing the consequences from physical attacks (see Chapter 4 ). While AT Planner has been validated against actual blast test data for nonwater systems, there remains significant uncertainty in the applicability of the modeling for water security because it has not been validated

against the structures specific to those systems. Therefore, the ongoing evaluation of AT Planner by the EPA and select water utility operators should include an assessment of the applicability of AT Planner for each of the critical and high-consequence components of a water system. The EPA and water utilities should then consider whether any additional validation testing is needed to determine specific failure modes of relevant water system components (e.g., actual storage tanks, pumps, water conduits, chlorine tanks) and possible countermeasures.

Summary of Research Priorities for Secure and Resilient Systems

Short-term priorities.

Develop an improved understanding of physical, cyber, and contaminant threats to water and wastewater systems, especially focusing on physical and cyber threats.

Communicate information on threats and consequences to water system managers through training and information exchange.

Develop an improved threat assessment procedure for water and wastewater utilities that will assist local utilities with their security and response planning.

Develop a process to assist local utilities in determining the consequences from physical, cyber, and contaminant attacks.

Update the risk assessment methodology for water systems to incorporate the latest approaches used in other industries, including developing credible threat descriptions and identifying cascading consequences.

Long-Term Priorities

Develop innovative design strategies for drinking water and wastewater systems that mitigate security risks and identify their costs and benefits in the context of public health, sustainability, cost-effectiveness, and homeland security. These designs might include:

In-pipe intervention strategies for drinking water systems,

Disaggregation of water and wastewater treatment facilities to achieve dual-use benefits, and

Designs that allow for interconnections and isolation.

Evaluate the need to validate AT Planner against structures specific to water systems.

Periodically review the EPA’s prioritized list of threats, contaminants, and threat scenarios to identify items that should be added to the list and remove items that are no longer a concern.

Continue development of technology transfer/training programs so that utilities understand the value of the EPA’s products for both homeland security incidents and natural disasters and know how to utilize the tools to their full extent.

Implementation of Priorities

Some of the research recommendations to support more resilient design and operation of drinking water and wastewater systems lie outside of the EPA’s traditional areas of expertise. To support the Action Plan efforts so far, the EPA has relied heavily on expert contractors to conduct this type of work. The EPA should continue to seek the relevant expertise of other federal agencies and national laboratories in these future efforts. However, the EPA will need to consider how best to balance intramural and extramural research funding to carry out this research, while maintaining appropriate oversight and input into the research activities (see also Chapter 5 ). Increasing staff expertise in some key areas, such as physical security, will be necessary to build a strong and well-rounded water security research program to support more resilient system design and operation.

RESEARCH RECOMMENDATIONS: IMPROVE THE ABILITY OF OPERATORS AND RESPONDERS TO DETECT AND ASSESS INCIDENTS

Suggestions are provided in this section for future research that should improve the ability of operators and responders to detect and assess water security incidents. Specific research suggestions in the areas of analytical methodologies and monitoring and distribution system modeling are discussed below.

Analytical Methodologies and Monitoring

Expanding existing analytical methods.

For some analytes of relevance to water security concerns, the available or approved detection methods are poor (e.g., some nonregulated analytes). More work needs to be done to expand existing methods to a broader range of analytes. For example, method 300.1 (EPA, 2000) covers only the common anions but could be extended to others, including toxic substances. The extension of existing methods to new analytes would allow a broader range of laboratories to expand their capabilities into the water security area.

Screening methods using conventional gas chromatography (GC) or high-performance liquid chromatography (HPLC) should also be investigated. Modern high-resolution chromatography combined with high-sensitivity detection (e.g., electron capture, fluorescence) is a powerful, yet accessible tool. Protocols should be developed to make the best use of these widely available capabilities. Software will have to be developed to facilitate the documentation of normal, background signals (fingerprint-type chromatograms). This background information can then be used to detect anomalies. Final protocols would have to be tested thoroughly against priority chemical contaminants. Chromatographic finger-prints have been used to monitor water supplies for nonintentional contamination, so this line of research would provide a dual benefit (D. Metz, Ohio River, personal communication, 2006; P. Schulhof, Seine River, personal communication, 2006).

Progress is being made with the protocol to concentrate samples and identify biological contaminants by polymerase chain reaction (PCR) analysis. Continued research, however, needs to be directed towards reducing the time and effort required to collect, process, and identify samples by automating portions of the protocol such as the concentration step. Such automated collection and sample processing systems would be especially valuable in response to security threats, when water samples could be channeled to existing or new detection technologies capable of onsite processing. The EPA should continue to expand the number of biothreat agents tested with the concentration/PCR protocol to include microbes other than spores, prioritizing test organisms that are both a threat to public health and resistant to chlorine (Morales-Morales, et al., 2003; Straub and Chandler, 2003). Continued testing of the concentration/PCR protocol should include various mixed suspensions of a target

microbe and background microbes to determine specificity of detection and various dilutions of the target microbe to determine sensitivity of detection. The protocol should also be tested on chloraminated water samples.

Developing New Monitoring Technologies

Chemical Detection. New chemical monitoring technologies for security-relevant analytes should be investigated. Examples include quartz crystal microbalance (QCM) sensors, microfluidic devices (lab-on-a-chip), ion-sensitive field-effect transistors (ISFETs), and larger-scale optrodes. Extramural agency and corporate partnerships developed by the EPA and longer-term research projects will help the evaluation and consideration of a broader range of detection platforms.

Biological Detection. Biological monitoring devices are essential to assess the type and extent of contamination in a suspected water security event. A broader range of innovative and developing detection technologies for biological agents, including methods that are field deployable and reagent-free, should be considered and evaluated. Innovative, field-deployable detection technologies (e.g., genetic fingerprinting, immunodetection, other technologies in development by universities, the Department of Defense, and industry) could reduce the time and effort for detection and enable earlier response efforts (Iqbal et al., 2000; Ivnitski et al., 2003; Lim et al., 2005; Monk and Walt, 2004; Yu and Bruno, 1996; Zhu et al., 2004). These new technologies might also increase the accuracy of detecting deliberate contamination events and reduce false alarms. Methods that can detect multiple biological agents and those with dual-use benefits should be emphasized over those methods limited to very specific agents (Peruski and Peruski, 2003; Rogers and Mulchandani, 1998). For example, DNA fingerprinting might be more useful than immunodetection systems dependent on a highly specific antibody for operation. The accuracy of these detection methods will depend on availability of quality reagents such as antibodies and primers; therefore, researchers will need to work closely with the Centers for Disease Control and Prevention (CDC) and other agencies that have access to such reagents.

Monitoring Devices for Wastewater Collection Systems . Contamination incidents have the potential to disrupt wastewater biological treat-

ment systems; thus, a long-term research program should also include research on monitoring technologies relevant to wastewater security concerns. Although a number of devices are available that can be used to monitor physical, chemical, and biological parameters, none of the currently available devices are robust or reliable enough when used in untreated wastewater to meet security requirements. The EPA should, therefore, encourage development of robust or reliable monitoring devices for wastewater infrastructure.

Syndromic Surveillance Tools. Syndromic surveillance tools may have the potential for detecting disease outbreaks and for investigating the possible role of water in such outbreaks (Berger et al., 2006). The EPA is already working to test two syndromic disease surveillance tools (RODS, ESSENCE) against prior water contamination outbreak data. There are substantive research needs that should be undertaken, however. Clearly, the improvement of existing syndromic surveillance tools is a long-term research objective. For syndromic surveillance to become worthwhile, it should achieve a favorable cost-benefit ratio considering the costs of false positives, and syndromic surveillance should also be adequately integrated into response plans. The implementation of syndromic surveillance systems on a large scale would require a more detailed linkage between disparate databases used in the public health sector and the water supply sector. Research to develop tools to allow local systems to readily fuse information from these disparate sources would be desirable. Such linkages would improve detection and response to waterborne disease outbreaks and more rapidly exclude water as a possible vehicle of disease. This would have important applications for both intentional and nonintentional water contamination events.

Real-Time Monitoring Systems

The development of a fully functional, easy-to-maintain, real-time monitoring system (RTMS) that could someday be used to prevent harm from deliberate attacks on the water system (“detect to prevent”), even with substantial research investments, is many years away. Therefore, the primary emphasis of future research on RTMSs, at least in the near term, should be on developing these technologies to assess the spread of contaminants, not to prevent exposure.

The committee also questions the likelihood of implementation of real-time monitoring devices for specific chemical or biological parame-

ters that are not useful in the day-to-day operation of a system (see Chapters 2 and 4 ). However, there are a few scenarios where implementation of continuous monitors for biological contaminants might be valuable, such as their use in certain water systems under heightened threat conditions (e.g., utilities for which specific intelligence information indicates they may be targeted). As discussed in Chapter 4 , deployment under these circumstances has a greater likelihood for success because the probability of an event is estimated to be much higher and the length of monitoring time is shortened. The use of highly sensitive and specific detection devices under such targeted circumstances would significantly lower the probability of false alarms and reduce the problem of poor positive predictive value (see Chapter 2 ) while also minimizing implementation and maintenance costs. Thus, improving monitoring systems for specific chemical or biological agents in drinking water is a valid long-term research goal. The EPA may find that longer-term research on more speculative sensor development could benefit from a further broadening of the circle of collaborators. Such speculative research may be more appropriately funded through the National Science Foundation or the Homeland Security Advanced Research Projects Agency, thus freeing up EPA resources for other purposes. To encourage such research, the EPA may wish to build its connections with the private sector on this technology.

Research on detection methods for RTMSs should proceed with careful consideration of the likelihood of implementation of the monitoring devices. In its near-term research plans, the EPA should adopt a first-stage approach to RTMSs, emphasizing generic sensors to detect intrusion or a system anomaly. The intrusion detection would then trigger more resource-intensive follow-up monitoring and analysis. Such an approach has significant dual-use benefits for routine contamination events that could outweigh the costs of implementing and operating these systems. Additional effort to develop cheaper, more accurate, and more easily deployable and maintainable sensors for routine water quality parameters would be useful both for anomaly detection and routine operation. Additional research is also needed, even in first-stage RTMSs, to understand normal water quality variations and distinguish variations that might be caused by a deliberate contamination attack. For example, continuous monitoring of chlorine residual at multiple points in the distribution system often reveals wide variations at different temporal scales due to changes in water demand that affect water residence time (e.g., operation of storage tanks). Although some work to understand inherent water quality variability in distribution systems is being conducted through the

Water Sentinel program, a significant amount of work is needed to translate the findings of this research into criteria for RTMSs to develop systems that have a reasonable likelihood of implementation.

An important component of RTMS research should include data fusion, whereby multiple anomalies must occur before an alarm signal is sent (see also Chapter 4 ). The private sector seems to be taking the lead on many types of multiparameter approaches to RTMSs and the processing of data, especially as described by contaminant or event signatures. It is important that the algorithms are open to peer review and can be accessed by all for development of new and refined approaches.

RTMS sensor research should consider a broader range of technologies, including full-spectrum UV and visible absorption, fluorescence excitation emission matrices, and ionization sensors (Alupoaei et al., 2004; Fenselau and Demirev, 2001; Lay, 2001). Many of these techniques are used as nonspecific chromatography detectors, and as such, they are highly sensitive. Most prototype RTMSs are composed of existing sensors that are designed to measure a specific contaminant, and some technologies have been excluded because they have not led to sensors with a high degree of selectivity. However, RTMSs need not be contaminant-specific; they only need to detect anomalies. Detection of an anomaly can then be followed by more specific contaminant analyses.

The problem of false positive signals from real-time contaminant-specific warning systems has been discussed in Chapter 2 . In essence, the problem is one of unfavorable arithmetic when the probability of a true positive is very small, as it would be for an intentional contamination attack on any particular water system of the tens of thousands of such systems. Therefore, most contaminant-specific alarm signals will be false positives. The EPA should consider the consequences of various rates of false positive signals for both large and small utilities and collect information on how alarms are currently handled by utilities. Workshops and structured surveys on this issue would provide valuable information on current practices, the extent to which positive signals are confirmed, the costs of false alarms, and the views of utility operators on their tolerance for various levels and types of false alarms. This research would provide useful guidance for the developers of water quality monitoring devices, for utilities that are considering implementing devices that are commercially available, and for local and state regulatory agencies who will need assistance interpreting alarm signals in light of the public health consequences.

Technology Testing

The EPA has developed a rigorous technology testing program to provide security product guidance to end users focusing on monitoring and decontamination technology. However, as noted in Chapter 4 , the number of relevant security technologies and agents of interest exceed the capacity and budget of the Technology Testing and Evaluation Program (TTEP). Therefore, developing a test-prioritization plan for TTEP seems especially important and is strongly recommended. Although the process of identifying technologies of interest has begun through the use of stakeholder meetings and advisory boards, activities to date have been weighted toward doing the easiest things first, and only some of these tests provided dual-use benefits. Balancing the homeland security benefits and the benefits to routine water system operations in TTEP will likely require additional strategic planning. One strategy has been to test equipment that is commercially available regardless of whether it addresses a high-risk agent. Instead, the EPA should look beyond the easy-to-identify commercially available equipment and make a greater effort to identify technologies in development that have the potential to address those agents identified as posing the greatest risk to water, considering the likelihood of the threat (including the ease of acquiring particular chemical or biological agents), the potential consequences, and the likelihood of implementing the technology. For a few of the highest-priority threats, the EPA may wish to consider providing technical support and/or funding to encourage more rapid development of a particularly promising technology that has a high likelihood of implementation and significant dual-use benefits, similar to the EPA Superfund Innovative Technology Evaluation (SITE) Emerging Technology Program.

Develop Laboratory Capability and Capacity

Adequate laboratory capacity is critical for responding to a terrorist incident affecting water supplies, and although this is not a research issue, the EPA has much to contribute from an applied perspective. The need for mobile analysis units capable of supplementing local laboratories and rapidly responding to geographical areas impacted by terrorist events should be considered. Such mobile laboratories could also address analytical needs that arise during natural catastrophes, such as Hurricane Katrina. Many states have begun to develop mobile laboratory

capabilities as part of their water security activities, and the EPA could glean information on their experiences to date.

The EPA is working with utilities and state and federal agencies to build a national laboratory response network for water sample analysis (i.e., the Water Laboratory Alliance). Some university laboratories may have capabilities that could merit inclusion in the nationwide network. Other laboratories may be stimulated to conduct additional research on improved analytical methods for toxic and biothreat agents if they were better informed of the current state of knowledge and had access to reference standards (access to some reference standards is currently limited due to security concerns). To be successful, a dual-use philosophy should be adopted whenever possible in the development of laboratory capacity (e.g., employing methods/instruments that can also be used for standard analytes).

Distribution System Modeling Tools

Distribution system models provide valuable tools for locating the source of contamination or assessing the spread if the source is known, estimating exposure, identifying locations for sampling, and developing decontamination strategies (see also Chapter 4 ). Distribution system models also have important dual-use applications to routine water quality concerns, and the EPA should continue to emphasize the dual-use value of its modeling tools. Specific recommendations are provided below to advance the capabilities and implementation of the Threat Ensemble Vulnerability Assessment (TEVA) and EPANET models.

Experimental Verification of Species Interaction Subcomponent Models

The final goal of producing a more flexible EPANET model through Multi-Species EPANET (MS-EPANET) is commendable. However, the new subcomponents are based upon developing better fundamental knowledge of reactions within the distribution system involving chemistry (e.g., disinfection kinetics, chemical partitioning), biology (e.g., development of biofilms, release and attachment of microbes), and materials science (e.g., corrosion of pipe materials and its relationship to disinfection efficacy). The large number of system constants in both MS-EPANET and TEVA necessitate significant investment in sensitivity

analysis research to quantify the accuracy of model predictions. The development and testing of all new features of MS-EPANET should be a long-term research goal. Until the validity of these subcomponents is verified and system constants can be assigned with more certainty, the water industry will be reluctant to use the full capability of MS-EPANET. Limitations in the accuracy of model predictions will need to be addressed in guidance to decision makers. A significant commitment will be needed in resources for experimental verification.

Alternate Approaches to Uncertainty Modeling

The Action Plan acknowledges correctly that the distribution system model simulations should incorporate an analysis of uncertainty because the point of attack is unknown. This has led to the use of the well-known Monte Carlo analysis to randomize the location of the attack and run repeated distribution system model simulations (1,000 or more) to generate a probability distribution to relate point of attack to human exposure impact. The focus on short-term results, however, has produced weaknesses in the current EPA approach to uncertainty research.

A broader discussion about how to incorporate uncertainty into the TEVA model should be invited. Approaches such as fuzzy logic (McKone and Deshpande, 2005) and Bayesian Maximum Entropy modeling (Serre and Christakos, 1999) are showing promise but have been applied mainly to homogenous space rather than to network domains. The EPA should encourage alternative ideas for handling uncertainty. If the expertise is not available within the agency, there needs to be a mechanism to expand extramural support for research, particularly within the university community.

Technology Transfer and Training in Use of the TEVA and EPANET Models

Advances in the TEVA model add significant complexity to the EPANET model, which may limit its widespread implementation. The EPA should work to communicate the capabilities of EPANET, MS-EPANET, and TEVA to utilities, emphasizing their value for routine water quality concerns, advanced homeland security planning, and contamination assessment and response activities. Until TEVA and MS-EPANET are further developed and widely available, the EPA should

consider an interim strategy to better inform water utilities on the value and use of existing distribution system models, such as EPANET. Progressive water utilities are already using EPANET to examine possible locations of attack and to track the concentration of contaminants within the distribution system.

Training in the use of MS-EPANET and the proposed TEVA model is also needed. Water utility managers need to be convinced that the costs for adapting a new model for their respective distribution systems are worthwhile, because many utilities have already invested heavily in development, verification, and calibration of existing models. The complexity of the TEVA model may increase these costs further, because many more implementation steps follow those for EPANET to adapt the TEVA “template” to the specifics of each water utility.

Summary of Research Priorities for Better Equipping Operators to Detect and Assess Incidents

Automate the concentration step of the concentration/PCR protocol.

Continue to test the concentration/PCR protocol:

Expand the number of biothreat agents tested to four or five organisms that include microbes other than spores, focusing on microbes that are both a threat to public health and resistant to chlorine.

Test the concentration/PCR protocol with chloraminated water samples.

Test the concentration/PCR protocol to determine sensitivity and specificity of detection.

Field-test RTMSs to determine false positive/false negative rates and maintenance requirements and develop basic criteria for the technology that might lead to a reasonable likelihood of implementation.

Continue research to develop a first-stage RTMS based on routine water quality sensors with dual-use applications.

Analyze the consequences of false positive signals from realtime monitoring systems, emphasizing current practices, the extent to which positive signals are confirmed, the costs of false alarms, and the tolerance of utility operators for false alarms.

Test standard chromatographic methods for their ability to screen for a broad range of toxic agents in routine laboratory testing.

Develop a test-prioritization strategy for TTEP to optimize the resources devoted to this effort.

Invite external peer review of the TEVA model before investing in field testing.

Long-term Priorities

Continue to develop portable, field-deployable systems that can be used to collect and process samples at event locations.

Formulate protocols and develop software for using GC- and HPLC-based fingerprinting to detect suspicious anomalies.

Stimulate research and ultimately development of new sensors for water security analytes based on innovative technologies, such as QCM, ISFETS, and microfluidics.

Evaluate and develop new field-deployable detection technologies for biological agents, including genetic fingerprinting, immunodetection, and reagentless technologies, that have the necessary sensitivity, specificity, and multiplex capabilities.

Develop improved, cheaper, and accurate RTMSs for routine water quality measurements.

Examine the use of nonspecific detection technologies for RTMSs.

Develop data fusion approaches for RTMSs that can minimize false positives.

Develop and test new monitoring technologies suitable for wastewater security applications.

Improve syndromic surveillance tools and develop a health surveillance network with appropriate linkages to water quality monitoring.

Continue to develop and refine the efficiency of a system-wide laboratory response network, including the development of mobile analysis units.

Continue fundamental research to understand the chemical and biological reactions that affect the fate and transport of contaminants in distribution systems to verify the constants used in MS-EPANET and TEVA.

Include alternative approaches to uncertainty design (e.g., fuzzy logic, Bayesian Maximum Entropy) in the TEVA model that are based more strongly upon stochastic than deterministic principles given that many of the input parameters to the current TEVA model are highly uncertain.

Develop projects for training water utilities in the value and use of EPANET, MS-EPANET, and TEVA.

Some of these research priorities may be more appropriately accomplished by universities, companies, or other agencies that have the necessary expertise, resources, and funding to successfully complete these tasks. The development of multiplex detection protocols and portable, field-deployable platforms are examples of tasks that might be better managed by some group other than the EPA. Work to determine the sensitivity and specificity of designated protocols for different biothreat agents could be conducted by university laboratories or private industry, with collaborative input from the EPA, considering their understanding of the needs of the water sector. Utilization of research resources outside the EPA would expand the variety of emerging, innovative analytical technologies that might be used to support the EPA’s efforts in enhancing the nation’s water security.

RESEARCH RECOMMENDATIONS: IMPROVED RESPONSE AND RECOVERY

Recommendations are provided in this section for future research that should improve response and recovery after a water security incident. Research suggestions related to tools and data for emergency planning and response, contingencies, risk communication and behavioral sciences, decontamination, and lessons learned from natural disasters are presented below.

Tools and Data for Emergency Planning and Response

Continued development of emergency response databases.

The EPA released preliminary versions of the Water Contamination Information Tool (WCIT) and the Consequence Assessment Tool (CAT) to provide data on contaminant properties, toxicity, and exposure threats (see Chapter 4 ), but the databases are still in their infancy, and numerous data gaps exist. The EPA will need to prioritize its continued efforts to further develop these response databases. Therefore, the EPA should develop strategic plans for WCIT and CAT, outlining the long-term goals for the databases and addressing questions such as:

What stakeholders will be served by the databases?

What categories of information do these stakeholders need?

How many contaminants should be included?

What linkages to other databases should be established?

The EPA will need to determine criteria for prioritizing what contaminants are added to the database and how to maintain and update the information. If WCIT and CAT are not continually revised to incorporate the latest scientific knowledge, the databases will become outdated. Expanding or even maintaining a database requires considerable resources, both intellectual and financial. If a commitment is not made initially for the necessary resources to update and maintain a database, spending the resources to create it becomes debatable. The EPA is currently facing similar issues maintaining its Integrated Risk Information System (IRIS) database.

The EPA should also clearly define the data quality objectives for WCIT/CAT and incorporate peer review of the data, as necessary, to meet these objectives. For example, the EPA may decide that some information about a contaminant is better than none, even if that information has limitations. This is a legitimate approach; however, the EPA should provide a mechanism that helps to ensure that individuals using the databases understand the data quality and their limitations. One mechanism for accomplishing this would be to add quality notations for each datum. Regardless of the approach taken, the EPA needs to describe the extent to which the data have been reviewed.

Evaluation and Improvement of Tools and Databases

With the forthcoming completion of at least the first stages of many tools and databases (e.g., WCIT, CAT), the EPA should consider the evaluation/improvement cycle. This will require the development of procedures to evaluate the utility and usability of these tools by potential constituencies. In addition, the EPA should take advantage of the tests afforded in response to “real-life” incidents. For example, some of the tools and databases were used (albeit in an early stage of their development) in the response to Hurricane Katrina. A formal assessment of knowledge gained from this experience could assist in the improvement and development of the tools.

Filling Data Gaps

The state of knowledge of the health risks from water contaminants that could be used in a malicious event is quite limited, as shown by the limited number of chemicals and even fewer biologicals in the WCIT/CAT databases and the many blank data fields in these databases. Important experimental and computational research is under way at the EPA to address some of these data gaps (see Chapter 4 , Section 3.6), but many gaps remain. There are two applications of toxicity/infectivity information that would be useful to the EPA for response and recovery efforts. The first is development of guidance for dissolved concentrations that would pose an immediate acute risk to exposed individuals, analogous to the inhalation immediate danger to life and health values of the National Institute for Occupational Safety and Health. The EPA is currently working on this problem by developing a database on acute and

chronic health effects associated with priority contaminants, although much work remains to be done. The second is guidance for determining the appropriate “acceptable” level remaining after cleanup/decontamination. This second aspect has not yet been strongly emphasized in the EPA research program. It is recommended that the EPA convene a working group to develop research and prioritization strategies for filling these data gaps and for ascertaining current gaps in knowledge with respect to rapid estimation of toxicity/infectivity in the absence of specific experimental information. Decisions for setting priorities for the data gathering efforts should be made with full consideration of dual-use benefits.

Contingencies for Water System Emergencies

Further study of water supply alternatives should be a high priority, considering their pivotal role in response and recovery and their dual-use applications for natural disasters or system failures. However, the subject of water supply contingencies seems to have been given a low priority in the EPA’s research program to date. Completion of the work in progress should be the first priority. The committee debated the value of investing significant resources in developing technologies that could supply drinking water for large communities over long-term disruptions because of the rarity of the need for such technologies. Nevertheless, the EPA should draw upon the research and development efforts of the Department of Defense in this area and work to test the application of these technologies to water security scenarios.

The EPA should consider including new research on contingencies for failures of the human subsystem in water system security. Such research could examine current practices for identifying back-up operators in the case of widespread incapacitation in both short-term and long-term scenarios. This research could also identify best practices, which could be incorporated into EPA guidance to water utilities for their emergency response planning.

Preliminary research suggests that geographic information systems (GIS) could be of significant value to utilities for identifying contingencies in the event of system failures. Therefore, further efforts may be needed to inform utilities about the value of GIS for emergency response and provide guidance for integrating GIS into their emergency planning procedures. National geodata standards may be needed to promote consistency and facilitate data exchange among users.

Behavioral Sciences and Risk Communication

The National Homeland Security Research Center (NHSRC) has made substantial progress in the development of risk communication guidance and training (see Chapter 4 ), but very little emphasis has been devoted to research on understanding how the public may respond to risk communication messages and how to improve communication of risks to the public. Terrorism presents risks that are new, evolving, and difficult to characterize; thus, water security poses communication challenges that should be addressed using scientifically rigorous research in the fields of risk communication and behavioral sciences. The EPA should continually reassess the role risk communication has in its overall risk management framework and fully integrate risk communication efforts into the overall risk management program. Behavioral science and associated risk communication research should be a high priority in the EPA’s future water security research plans. The following recommendations are targeted toward water-security events, but the proposed research has dual benefits for improving non-security-related communications with the public.

Analysis of Factors that Build Trust and Improve Communication

Research and experience prove that one of the most important keys to communication success is an organization’s ability to establish, maintain, and increase trust and credibility with key stakeholders, including employees, regulatory agencies, citizen groups, the public, and the media. To improve overall communication strategies in a water-related emergency, research is needed that analyzes factors that build trust and reduce fear (e.g., What types of concerns do people have related to public health emergencies, water security issues, or bioterrorism? How do utilities build trust and credibility with the public around water security incidents?). In addition, research is needed to analyze methods to counter and reduce the possibility of misinformation or false information being distributed to the public and key stakeholders.

Understanding Institutional Behavior

Building response and recovery capacity requires agencies that might be involved in a water security event to develop stronger working relationships. Although water utilities, public health agencies, law enforcement, emergency responders, and the media do not have a long history of collaborating and working together, several state drinking water programs have taken the lead in carrying out tabletop exercises as well as on-the-ground exercises to address this issue. These state programs have also undertaken measures to facilitate an understanding of the roles and responsibilities of the various potential players, including federal, state, and local law enforcement; state and local health agencies; state and local emergency response agencies; and water utilities. The EPA could glean useful information from these ongoing state and local activities. Nevertheless, additional research is needed to better understand the culture of the agencies that will be responding to events, how these agencies will interact in a water-related crisis, and what level of effort is needed to maintain collaboration in planning and preparedness. This research could identify barriers to more effective collaboration, and these findings could be used to create training scenarios that could improve coordination and resolve potential conflicts in advance. This research is a short-term priority given the importance of coordinated interaction during a crisis. The research could be performed relatively quickly because there is a wealth of experiences, particularly at the state level, related to agency interactions in water-related crises.

Investigate Applicability of Research in Behavioral Science

While some of the recommended research on risk communication and behavioral science may need to be managed by the EPA to address specific water security-related issues, the EPA should also take advantage of other behavioral science research currently being conducted through university-based partnerships, including those established by the Homeland Security Centers of Excellence program. For example, the University of Maryland’s National Consortium for the Study of Terrorism and Responses to Terror (START) is conducting original research on issues that are poorly understood, including risk perception and communication, household and community preparedness for terrorist attacks, likely behavioral responses by the public, social and psychological vulnerability to terrorism, and strategies for mitigating negative psychologi-

cal effects and enhancing resilience in the face of the terror threat. The START center is also synthesizing existing research findings in order to provide timely guidance for decision makers and the public, paying special attention to how diverse audiences react to and are affected by threats and preparedness efforts.

In addition, the CDC has developed a national network of 50 Centers for Public Health Preparedness (CPHP) to train the public health workforce to respond to threats to our nation's health, including bioterrorism. These centers work to strengthen terrorism preparedness and emergency public health response at the state and local level and to develop a network of academic-based programs contributing to national terrorism preparedness and emergency response capacity. Information from the CPHP may be relevant and useful to the water sector.

Pretesting Risk Communication Messages

Although the message mapping workshops are a good start to assist stakeholders in preparing messages that will be relevant in a water security incident, the messages have not been tested and evaluated. Therefore, the EPA should engage the research community in pretesting messages being developed by the Center for Risk Communication so that case studies and scenarios can be analyzed for effectiveness in reaching key audiences, and problems can be corrected in advance. Sophisticated evaluation techniques and standard research procedures are used by the CDC to pretest public messages. This evaluation research should be based on standard criteria established in the risk communication literature (e.g., Mailback and Parrott, 1995; National Cancer Institute, 2002; Witte et al., 2001).

Analysis of the Risks and Benefits of Releasing Security Information

The decision of when to release or withhold water security information is critical to the development of a risk communication strategy. Therefore, the EPA should analyze the risks and benefits of releasing water security information, considering input from its broad range of constituents, and develop transparent agency guidance on when to release information versus when to withhold it due to security concerns.

The committee considers this a priority because of the difficulty and importance of the information sharing problem.

Water-Related Risk Communication Training

As the lead U.S. agency in water system security, the EPA should assume the responsibility for developing a national training program on water-related risk communication planning and implementation for water managers. This should be done in collaboration with the water and wastewater organizations, state government agencies, public health officials, health care officials, and others engaged in communication of risks during water-related emergencies.

Decontamination

Decontamination research is critical to improving response and recovery, and the products are applicable to address unintentional contamination events from natural disasters (e.g., hurricanes, floods, earthquakes) and routine malfunctions (e.g., pipe breaks, negative pressures due to power losses). The EPA has numerous ongoing projects in this area that should be completed, but additional research topics are also suggested below.

Addressing Data Gaps

EPA decontamination research products released thus far have shown that fundamental physical, chemical, and/or biological characteristics of many threat agents of concern are not yet known. Therefore, additional laboratory research is needed related to the behavior of contaminants in water supply and wastewater systems and methods for decontaminating water infrastructure. For example, one research priority would be to develop inactivation rate data for all microbes of concern with both free and combined chlorine strategies, because both approaches are used in the water industry. Rate and equilibrium data for adsorption/desorption of contaminants on pipe walls is also needed, although the EPA could also take advantage of existing databases on structure-activity relationships to predict these behaviors. Long-term re-

search, perhaps in partnership with other Office of Research and Development units, could enhance our understanding of the fate, transport, and transformation of toxics in water and wastewater environments.

Decontamination Strategies

The EPA should build on its ongoing work in the area of decontamination and address gaps in the current knowledge base. For example, research is needed to examine readily available household inactivation methods for biological agents (including spore-formers), such as microwaving. The EPA should also work to further the development of innovative decontamination technologies that address important water security concerns. Research and development on new POU/POE technologies, such as superheated water devices, could help overcome operational disadvantages of the products currently on the market.

Prioritizing Future Surrogate Research

Surrogates are relevant to numerous water security research applications, including research on contaminant fate and transport, human exposure risks, and decontamination. Research is ongoing to identify surrogates or simulants for biological agents, to determine which surrogates are appropriate, and to determine the ability of typical drinking water disinfection practices (chlorination and chloramination) to inactivate those agents (see Chapter 4 , Section 3.2). Much of the research has focused on Bacillus anthracis and other bacterial agents, but the EPA should determine if surrogates for research on biotoxins and viruses are needed and whether additional surrogates are needed for other bacterial agents. A viral simulant or surrogate would be helpful to examine virus survival in fresh water, drinking water, and sewage, as well as virus susceptibility to water disinfectants. Research in this area has relevance to viral bioterrorism agents and also has strong dual-use research applications because viral surrogates could facilitate risk assessment studies on natural viruses (e.g., SARS, avian influenza).

Surrogate research is a laborious experimental process (see Box 4-1 ) that must be conducted in one of the few laboratories already authorized to keep and work with select agents. Considerable research is required to compare the select agent with candidate surrogates under the experimental conditions of interest. As discussed in Chapter 4 , surrogates need not

mimic in all respects the agents they stand in for. For some important security or decontamination uses, it may only be necessary that they provide an appropriate bound on the characteristic of interest in the target agent (e.g., persistence, disinfectant sensitivity). Therefore, the EPA should carefully consider and prioritize the agents and the research applications for which surrogates are needed. The prioritization process for surrogates should consider the following:

Which types of research could be greatly facilitated through the availability of surrogates?

Which types of research with surrogates might have “dual-use” applications (i.e., could the properties of certain surrogates also be usefully extrapolated to other common organisms)?

Which types of research should be done only with select agents?

How closely should the surrogate properties of interest match that of the target organism?

What are the costs and benefits to the research program associated with surrogate development versus use of the pathogenic agents?

The EPA should engage a limited number of individuals (e.g., federal partners, academics) who are involved in similar research in this prioritization process.

Lessons Learned from Natural Disasters

Midway through the committee’s work, NRC (2005; see Appendix A ) suggested the EPA take advantage of experience gained in the aftermath of Katrina so as to improve future response and recovery efforts for water security. While a hurricane caused this catastrophe, it is conceivable that a similar result might have occurred if the levees had been destroyed by terrorist explosives. Thus, New Orleans offered a living laboratory to study many aspects of the impacts of a disaster on water and wastewater systems of all sizes. Failure modes, infrastructure interdependencies, decontamination and service restoration strategies, the availability of alternative supplies, communication strategies, and the ability to service special institutions (e.g., hospitals) and special needs individuals could all have been examined in the immediate aftermath of the hurricane. To the best of the committee’s knowledge, however, the EPA has not attempted to compile a knowledge base from this experience. As

time passes, it will become increasingly difficult to reconstruct what transpired. Other natural or manmade disasters, such as the earthquakes in California in 1989 and 1994 or the “Great Flood of 1993” in the Mid-west, or natural contamination events, such as the Milwaukee C ryptosporidium outbreak, may also offer opportunities to mine important data about the failure or recovery of water and wastewater systems, but detailed information on these earlier occurrences may be lacking. In the future, the NHSRC should be poised to seize opportunities for learning about response and recovery after major natural or man-made disasters affecting water or wastewater systems.

Summary of Research Priorities for Improving Response and Recovery

Determine strategic plans for managing and maintaining the WCIT/CAT databases, considering the likely uses and long-term goals for the databases.

Develop and implement a strategy for evaluating the utility and usability of the response tools and databases, including stakeholder feedback and lessons learned during their use under “real-life” incidents.

Convene a working group to develop research strategies for filling the data gaps in WCIT/CAT and other planned emergency response databases.

Contingencies for Water Emergencies

Complete the work in progress on contingencies and infrastructure interdependencies under Section 3.5 of the Action Plan.

Test and evaluate the most promising innovative water supply technologies that enable or enhance the short- or long-term delivery of drinking water in the event of systemic failure of water systems. Analyze the positive features and those areas needing improvement prior to full-scale deployment.

Conduct research on potential contingencies for failures of the “human subsystem.”

Analyze factors that build trust, reduce fear, and prevent panic to improve overall communication strategies in a water-related emergency.

Investigate the behavioral science research being conducted by the Homeland Security University Centers of Excellence and other federal agencies for applicability to the water sector.

Pretest messages being developed by the Center for Risk Communication and analyze case studies and scenarios for effectiveness.

Analyze the risks and benefits of releasing security information to inform the EPA’s risk communication strategies and its practices on information sharing.

Fully integrate risk communication efforts into the overall risk management program and provide adequate resources that ensure these efforts remain a high priority in the EPA’s future water security research program.

Conduct research to better understand how agencies will interact in a water-related crisis situation and determine what strategies will be most effective in encouraging and maintaining collaboration in planning and preparedness.

Complete the many decontamination projects in progress under Section 3.4 of the Action Plan.

Develop predictive models or laboratory data for inactivation of bioterrorism agents in both free chlorine and chloramines that can be used in MS-EPANET and the TEVA model.

Explore development and testing of new POU/POE devices that may overcome the disadvantages of existing devices.

Examine readily available household inactivation methods for biological agents (including spore-forming agents), such as microwaving.

Determine the costs and benefits of further research to identify additional surrogates, considering which agents under which conditions or applications should be prioritized for surrogate development research.

Use the remaining data from the experience of Hurricane Katrina to analyze the optimal response and recovery techniques (e.g., water supply alternatives, contingency planning, and infrastructure interdependencies) that would also apply to water security events.

Integrate experience with decontamination of the distribution system in New Orleans after Hurricane Katrina to improve EPA guidance for water security decontamination.

Evaluate risk communication strategies related to Hurricane Katrina or other past disaster events to determine if communication strategies related to drinking water safety reached the most vulnerable populations.

Develop a post-event strategy for learning from future natural disasters affecting water systems. This strategy should support on-site assessments of impacts and interdependencies and evaluations of successes and failures during response and recovery.

Continue to develop and maintain the WCIT/CAT databases according to the objectives set forth in the strategic database management plan. Incorporate a mechanism to provide on-going peer review of the data to meet its data quality objectives.

Continue experimental and computational research to fill critical data gaps in WCIT/CAT, including research on the health effects of both acute and chronic exposure to priority contaminants.

Develop new, innovative technologies for supplying drinking water to affected customers over both short- and long-term water system failures.

Risk Communication and Behavioral Sciences

Develop a program of interdisciplinary empirical research in behavioral sciences to better understand how to prepare stakeholders for water security incidents. The EPA should support original research that will help address critical knowledge gaps. For example:

What are the public’s beliefs, opinions, and knowledge about water security risks?

How do risk perception and other psychological factors affect responses to water-related events?

How can these risks be communicated more effectively to the public?

Develop a national training program on water-related risk communication planning and implementation for water managers.

Continue laboratory research to fill the data gaps related to behavior of contaminants in water supply and wastewater systems and methods for decontaminating water infrastructure.

Continue surrogate research based on the research prioritization determined in collaboration with an interagency working group. The EPA should also explore ways that this surrogate research could assist in responding to everyday agents or to other routes of exposure (e.g., inhalation, inactivating agents on surfaces).

The EPA has historically been a lead federal agency in understanding the fate and transport of contaminants in the environment and has a clear understanding of the practical concerns of the water sector. Thus, the EPA remains the appropriate lead agency to develop the tools for emergency response and to prioritize the research needed to fill the remaining gaps, with input from key stakeholders. The EPA is also well suited to develop a national training program on water-related risk communication and to evaluate lessons learned from Hurricane Katrina and other past disaster events. However, innovative technology development research, such as the development of novel technologies for supplying water during system failures, should be conducted by other agencies,

university researchers, or firms with the greatest expertise. The EPA, instead, should focus its efforts on harvesting information on existing technologies, synthesizing this information for end users, and providing guidance to developers on unique technology needs for water security. Behavioral science research and evaluation research is more appropriately conducted by universities or other federal agencies (e.g., CDC) that have the necessary expertise to complete these tasks. However, the EPA still needs in-house behavioral science experts able to supervise and use this work to best advantage.

CONCLUSIONS AND RECOMMENDATIONS

In this chapter, recommendations are provided for future research directions in the area of water security. Two key water security research gaps—behavioral science and innovative future system design—that were not considered in the short-term planning horizon of the Action Plan are identified. In accordance with the committee’s charge (see Chapter 1 ), short- and long-term water security research priorities are presented in three areas: (1) developing products to support more resilient design and operation of facilities and systems, (2) improving the ability of operators and responders to detect and assess incidents, and (3) improving response and recovery.

The EPA should develop a program of interdisciplinary empiri cal research in behavioral science to better understand how to pre pare stakeholders for water security incidents. The risks of terrorism are dynamic and uncertain and involve complex behavioral phenomena. The EPA should take advantage of existing behavioral science research that could be applied to water security issues to improve response and recovery efforts. At the same time, when gaps exist, the EPA should support rigorous empirical research that will help address, for example, what the public’s beliefs, opinions, and knowledge about water security risks are; how risk perception and other psychological factors affect responses to water-related events; and how to communicate these risks effectively to the public.

The EPA should take a leadership role in providing guidance for the planning, design, and implementation of new, more sustainable and resilient water and wastewater facilities for the 21st century. Given the investments necessary to upgrade and sustain the country’s water and wastewater systems, research on innovative approaches to make the infrastructure more sustainable and resilient both to routine and

malicious incidents would provide substantial dual-use benefits. The EPA should help develop and test new concepts, technologies, and management structures for water and wastewater utilities to meet objectives of public health, sustainability, cost-effectiveness, and homeland security. Specific research topics related to drinking water and wastewater, such as decentralized systems and in-pipe interventions to reduce exposure from contaminants, are suggested.

Recommended research topics in the area of supporting more resilient design and operation of drinking water and wastewater systems include improved processes for threat and consequence assessments and innovative designs for water and wastewater. A thorough and balanced threat assessment encompassing physical, cyber, and contaminant threats is lacking. To date, the EPA has focused its threat assessments on contaminant threats, but physical and cyber threats deserve more attention and analysis because this information could influence the EPA’s future research priorities and utilities’ preparedness and response planning.

Research suggestions that improve the ability of operators and responders to detect and assess incidents build upon the EPA’s current research in the areas of analytical methodologies and monitoring and distribution system modeling. In the short term, the EPA should continue research to develop and refine a first-stage RTMS based on routine water quality parameters with dual-use applications. Long-term research recommendations include the development of innovative detection technologies and cheaper, more accurate RTMSs. To support the simulation models in development, a substantial amount of fundamental research is needed to improve understanding of the fate and transport of contaminants in distribution systems. Based on the number of emerging technologies and agents of interest, the EPA should develop a prioritization strategy for technology testing to optimize the resources devoted to this effort.

Recommendations for future research priorities to improve response and recovery emphasize the sustainability of tools for emergency planning and response (e.g., WCIT/CAT) and improving research on water security contingencies, behavioral sciences, and risk communication. The EPA should also evaluate the relative importance of future laboratory work on surrogate development and address data gaps in the knowledge of decontamination processes and behavior. So far, the EPA has not taken advantage of the many opportunities from Hurricane Katrina to harvest lessons learned related to response and recovery, and the window of opportunity is rapidly closing.

Some of the research recommendations provided in this chapter lie outside of the EPA’s traditional areas of expertise. The EPA will need to consider how best to balance intramural and extramural research funding to carry out this research, while maintaining appropriate oversight and input into the research activities. Increasing staff expertise in some key areas, such as physical security and behavioral sciences, will be necessary to build a strong and well-rounded water security research program.

This page intentionally left blank.

Welcome to OpenBook!

You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

Do you want to take a quick tour of the OpenBook's features?

Show this book's table of contents , where you can jump to any chapter by name.

...or use these buttons to go back to the previous chapter or skip to the next one.

Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

Switch between the Original Pages , where you can read the report as it appeared in print, and Text Pages for the web version, where you can highlight and search the text.

To search the entire text of this book, type in your search term here and press Enter .

Share a link to this book page on your preferred social network or via email.

View our suggested citation for this chapter.

Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

Get Email Updates

Do you enjoy reading reports from the Academies online for free ? Sign up for email notifications and we'll let you know about new publications in your areas of interest when they're released.

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

• View all journals
• Explore content
• About the journal
• Publish with us
• Sign up for alerts
• NATURE INDEX
• 25 March 2024

Larger or longer grants unlikely to push senior scientists towards high-risk, high-reward work

• Dalmeet Singh Chawla 0

Dalmeet Singh Chawla is a freelance science journalist based in London.

You can also search for this author in PubMed   Google Scholar

The duration and value of a grant are not likely to alter the research strategies of recipients in the United States. Credit: DigitalVision/Getty

Offering professors more money or time isn’t likely to dramatically change how they do their research, a survey of US-based academics has found.

The survey, described in a preprint article posted on arXiv in December 1 , was completed by 4,175 professors across several disciplines, including the natural sciences, social sciences, engineering, mathematics and humanities.

The study’s authors, Kyle Myers and Wei Yang Tham, both economists at Harvard Business School in Boston, Massachusetts, say the aim was to investigate whether senior scientists would conduct their research differently if they had more money but less time, or vice versa.

The research comes amid interest from some funders in tweaking the amount of time and money awarded to scientists to incentivize them to do more socially valuable work. For instance, in 2017, the Howard Hughes Medical Institute in Chevy Chase, Maryland, announced that it had extended its grants from five to seven years, arguing that the extra time would allow researchers to “take more risk and achieve more transformative advances”.

Acknowledging that the most reliable way to test how grant characteristics might affect researchers’ work is to award them actual grants — which was not feasible — Myers and Tham instead presented them with hypothetical scenarios.

The survey respondents were asked what research strategies they would pursue if they were offered a certain sum of grant money for a fixed time period. Both the value and duration were randomly assigned. The hypothetical grants were worth US$100,000 to $2 million and ran between two and ten years.

To capture the changes in strategy, the survey provided the participants with five options that they could take if they successfully obtained the hypothetical grant. These included pursuing riskier projects — for example, those with only a small chance of success – or ones that were unrelated to their current work and increasing the speed or size of their ongoing projects.

The survey revealed that longer grants increased the researchers’ willingness to pursue riskier projects — but this held true only for tenured professors, who can afford to take a gamble because they tend to have long-term job security, an established reputation and access to more resources. The authors note, however, that any change in research strategy that resulted from receiving a longer grant was not substantial.

Non-tenured professors were not swayed towards risk-taking when they received longer grants. This finding suggests that longer grant designs don’t take into account the pressures that come with shorter employment contracts, says Myers. “If you’re a professor who’s on a 1- or 2-year contract, where you have to get renewed every year, then the difference between a 5-year or 10-year grant is not as important as performing in the next year or two,” he says.

Both tenured and non-tenured professors said longer, larger grants would slow down how fast they worked, “which suggests a significant amount of racing in science is in pursuit of resources”, the authors say, adding that this effect was also minor.

Myers and Tham report that the professors were “very unwilling” to reduce the amount of grant funding in exchange for a longer duration. “Money is much more valuable than time,” they conclude. They found that the professors valued a 1% increase in grant money nearly four times more than a 1% increase in grant duration. The study concludes that the researchers didn’t seem a to view the length of a single grant as “an important constraint on their research pursuits given their preferences, incentives and expected access to future funding sources”.

Experimenting with grant structures

Carl Bergstrom, a biologist at the University of Washington in Seattle who has studied science-funding models, says it’s interesting that substantial changes in grant structure generally yielded little to no change in the researchers’ hypothetical behaviour. “I just don’t know what to make of that,” he says, noting that it’s unclear whether this finding is a result of the study design, or is saying something about scientists’ attitude towards change. “One consistent explanation of all of this would be that fairly reasonable changes in the structure of one particular individual grant don’t do enough to change the overall incentive structure that scientists face for them to alter their behaviour.”

Bergstrom adds that modifying grant structures can still be a valuable exercise that could result in different kinds of candidate applying for and securing funding, which in turn might affect the kind of research that is produced. Myers and Tham didn’t examine whether modifying grant structures would affect the diversity of the pool of candidates, but they have investigated the nuances of risk-taking in research in another study, also posted as a preprint in December 2 . Researchers were surveyed about their appetite for risky science and how it affected their approach to grants. The survey found a strong link between the perceived risk of research and the amount of time spent applying for grants .

To get a clearer understanding of whether the findings of the surveys would hold in the real world, funders would need to modify actual grants, says Myers. He acknowledges that this would be a big commitment and a risk, but doing so could have significant benefits for science.

There is growing interest in finding more efficient and effective grant structures. In November, the national funder UK Research and Innovation launched a new Metascience Unit , which is dedicated to finding more sophisticated and efficient ways to make funding and policy decisions. The following month, the US National Science Foundation announced that it would be conducting a series of social and economic experiments to determine how its funding processes can be improved.

As for the survey, Myers hopes the findings can provide insights to inform such initiatives. “As long as we’ve reduced uncertainty about what is the best way forward, that is very valuable,” he says. “We hope that our hypothetical experiments are motivation for more real-world experiments in the future.”

doi: https://doi.org/10.1038/d41586-024-00929-5

Myers, K. & Tham, W. Y. Preprint at arXiv https://doi.org/10.48550/arXiv.2312.06479 (2023).

Myers, K. R. et al. Preprint at arXiv https://doi.org/10.48550/arXiv.2312.01442 (2023).

Download references

Related Articles

• Institutions
• Research management

A fresh start for the African Academy of Sciences

Editorial 19 MAR 24

Is the Mars rover’s rock collection worth $11 billion?

News 19 MAR 24

Bring PhD assessment into the twenty-first century

Editorial 12 MAR 24

China–US climate collaboration concerns as Xie and Kerry step down

News 12 MAR 24

Biden seeks to boost science funding — but his budget faces an ominous future

‘Despair’: Argentinian researchers protest as president begins dismantling science

News 07 MAR 24

Numbers highlight US dominance in clinical research

Nature Index 13 MAR 24

A spotlight on the stark imbalances of global health research

Postdoctoral Associate- Cellular Neuroscience

Houston, Texas (US)

Baylor College of Medicine (BCM)

Postdoctoral Associate- Cancer Biology

Recruitment of talent positions at shengjing hospital of china medical university.

Call for top experts and scholars in the field of science and technology.

Shenyang, Liaoning, China

Shengjing Hospital of China Medical University

Assistant Professor in Plant Biology

The Plant Science Program in the Biological and Environmental Science and Engineering (BESE) Division at King Abdullah University of Science and Te...

Saudi Arabia (SA)

King Abdullah University of Science and Technology

Postdoctoral Associate

Our laboratory at the Washington University in St. Louis is seeking a postdoctoral experimental biologist to study urogenital diseases and cancer.

Saint Louis, Missouri

Washington University School of Medicine Department of Medicine

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Quick links

• Explore articles by subject
• Guide to authors
• Editorial policies

• Open access
• Published: 21 March 2024

GRADE-ADOLOPMENT of hyperthyroidism treatment guidelines for a Pakistani context

• Russell Seth Martins 1   na1 ,
• Sarah Nadeem 1 , 2 , 3   na1 ,
• Abeer Aziz 1 ,
• Sajjan Raja 4 ,
• Alina Pervez 1 ,
• Najmul Islam 2 ,
• Asma Ahmed 2 ,
• Aisha Sheikh 2 ,
• Saira Furqan 2 ,
• Nanik Ram 2 ,
• Azra Rizwan 2 ,
• Nashia Ali Rizvi 1 ,
• Mohsin Ali Mustafa 1 ,
• Salima Saleem Aamdani 5 ,
• Bushra Ayub 6 &
• Muhammad Qamar Masood 2  

BMC Endocrine Disorders volume  24 , Article number:  41 ( 2024 ) Cite this article

116 Accesses

1 Altmetric

Metrics details

Introduction

The prevalence of hyperthyroidism in Pakistan is 2.9%, which is two times higher than in the United States. Most high-quality hyperthyroidism clinical practice guidelines (CPGs) used internationally originate from high-income countries in the West. Local CPGs in Pakistan are not backed by transparent methodologies. We aimed to produce comprehensive, high-quality CPGs for the management of hyperthyroidism in Pakistan.

We employed the GRADE-ADOLOPMENT approach utilizing the 2016 American Thyroid Association Guidelines for Diagnosis and Management of Hyperthyroidism and Other Causes of Thyrotoxicosis as the source CPG. Recommendations from the source guideline were either adopted as is, excluded, or adapted according to our local context.

The source guideline included a total of 124 recommendations, out of which 71 were adopted and 49 were excluded. 4 recommendations were carried forward for adaptation via the ETD process, with modifications being made to 2 of these. The first addressed the need for liver function tests (LFTs) amongst patients experiencing symptoms of hepatotoxicity while being treated with anti-thyroid drugs (ATDs). The second pertained to thyroid status testing post-treatment by radioactive iodine (RAI) therapy for Graves’ Disease (GD). Both adaptations centered around the judicious use of laboratory investigations to reduce costs of hyperthyroidism management.

Our newly developed hyperthyroidism CPGs for Pakistan contain two context-specific modifications that prioritize patients’ finances during the course of hyperthyroidism management and to limit the overuse of laboratory testing in a resource-constrained setting. Future research must investigate the cost-effectiveness and risk-benefit ratio of these modified recommendations.

Peer Review reports

Hyperthyroidism is a common endocrine condition that presents a significant global challenge with high morbidity and mortality rates [ 1 , 2 ]. In Pakistan, a South Asian lower-middle-income country (LMIC) with a population of over 220 million [ 3 ], the prevalence is 2.9% [ 4 ]. This is more than two times higher than the United States of America (US: 1.3% [ 5 ]) and more than three times higher than in Europe (0.8% [ 6 ]). The high prevalence of hyperthyroidism in Pakistan can be attributed to a complex interplay of factors, with key determinants including geographical variables and ethnic diversity [ 7 ]. Hyperthyroidism confers an increased all-cause mortality risk, particularly due to cardiovascular causes [ 8 ].

Evidence-based clinical practice guidelines (EBCPGs) direct the diagnosis and management of hyperthyroidism, so as to achieve standardization of favorable clinical outcomes [ 9 , 10 ]. EBCPGs created by institutions in developed countries in the West, such as the US [ 11 ] and European countries [ 12 ], are oftentimes adopted by other countries, particularly LMICs, for use in their settings. This is because LMICs, like Pakistan, usually lack the research infrastructure and financial resources to independently develop EBCPGs de novo for their own healthcare context [ 13 ]. However, the application of such EBCPGs for the management of hyperthyroidism in Pakistan presents a problem, as the country’s landscape differs due to several factors [ 14 ]. These include disease epidemiology [ 15 ], healthcare financing [ 16 ], dietary habits and iodine consumption [ 17 ], socio-economic influences [ 18 ], and disease-related awareness [ 19 ]. Therefore, it becomes imperative for an LMIC like Pakistan to create EBCPGs that best suit the unique context of the setting where they will be applied.

In cases where the de novo creation of EBCPGs is not practically feasible, a process called “adolopment” provides a suitable alternative. Adolopment describes a combination of adoption (verbatim use), adaptation (contextual modifications), and de novo development, thus leveraging the benefits of pre-existing high-quality EBCPGs while ensuring local appropriateness. The GRADE-ADOLOPMENT method [ 13 ], developed by GRADE (Grading of Recommendations Assessment, Development, and Evaluation), is a globally accepted and implemented process of EBCPG adolopment. It uses evidence-to-decision (ETD) tables, which summarize the best available evidence on a topic, to guide decisions regarding the need for contextual modifications of individual recommendations within an EBCPG [ 20 ]. GRADE-ADOLOPMENT has been used in countries and regions across the world, including Saudi Arabia [ 13 ], Australia [ 21 ], Tunisia [ 22 ], the Eastern Mediterranean region [ 23 ], the Asia-Pacific region [ 24 ], Mexico [ 25 ], and the United Kingdom [ 26 ].

Although the Pakistan Endocrine Society, founded in 2003, is involved in the creation of local EBCPGs for the management of common endocrine disorders in Pakistan, their publications have thus far focused on diabetes mellitus and metabolic syndrome [ 27 ]. Moreover, the processes involved in the development of these EBCGPs are not explicitly described. Consequently, there is immense need for local hyperthyroidism EBCPGs to be developed following a transparent, standardized process that makes use of existing available best-evidence EBCPGs with appropriate context-specific modifications. Such EBCPGs would bring the healthcare system of Pakistan a step closer to achieving optimal health outcomes in hyperthyroidism and would gain credibility by virtue of their transparent development processes. Thus, we aimed to employ the GRADE-ADOLOPMENT process to develop local evidence-based EBCPGs for the management of hyperthyroidism in adults by GPs in Pakistan.

Methodology

This process was conducted at the CITRIC (Clinical and Translational Research Incubator) Center for Clinical Best Practices (CCBP) at the Aga Khan University (AKU), Pakistan. The AKU is a private sector, not-for‐profit hospital in Pakistan, and is also the country’s leading healthcare and biomedical research facility [ 28 ].

The CITRIC CCBP at AKU is tasked with the adaptation and development of EBCPG and care pathways to standardize and improve healthcare in Pakistan. The GRADE-ADOLOPMENT processes described in this study have been implemented by the CCBP, in collaboration with the expertise of the Section of Endocrinology at AKU and the GRADE-USA working group, in the development of hyperthyroidism management EBCPGs for use by general practitioners (GPs)/primary care physicians in Pakistan. The decision to create hyperthyroidism EBCPGs for GPs rather than specialist endocrinologists is due to the lack of access to specialists in Pakistan [ 29 ].

The study team is comprised of the CCBP research staff (who are proficient in GRADE methodology and in the development of EBCPGs) as well as endocrinology faculty led by Endocrinology Section Head of AKU.

Source guideline selection

The source guideline is the single, original, “parent” EBCPG that undergoes the GRADE-ADOLOPMENT process in the development of a local EBCPG. The 2016 American Thyroid Association Guidelines for Diagnosis and Management of Hyperthyroidism and Other Causes of Thyrotoxicosis [ 30 ] was selected by the Section of Endocrinology as the source EBCPG, due to its comprehensive set of recommendations, integrated approach to management, and high-quality synthesis of available evidence. The 2016 American Thyroid Association source guideline used the GRADE approach for the strength of the recommendations and the quality of evidence.

Guideline review

Figure  1 delineates the adolopment process used in our study. First, a Table of Recommendations (ToR) was created by extracting and compiling all recommendations mentioned in the source EBCPG. Two senior attending endocrinologists reviewed the ToR independently and marked each recommendation as either “ Adopt ,” “ Adapt” or “ Exclude .” Discrepancies were settled in consensus with the Section Head of Endocrinology. Recommendations marked “ Adopt ” were incorporated as is or with minor changes into the local EBCPG, while those marked “ Exclude ” were omitted from the local EBCPG. Exclusion was based on the recommendation pertaining to pediatric or inpatient management, or if the recommendation was deemed irrelevant to the local Pakistani context. Other reasons for exclusion were required to be explained by the reviewers.

Recommendations marked “ Adapt ” were deemed to warrant additional review and revision via the GRADE-ADOLOPMENT process (detailed below) before incorporation into the local EBCPGs. Our adolopment process (Fig. 1 ) had two important differentiations to the one described originally [ 13 ]. Firstly, we did not create any recommendations de novo, which was due to a lack of perceived need for additional recommendations. Secondly, recommendations that were deemed to require only minor and straightforward changes prior to adoption were not subjected to the complete adaptation process consisting of ETD tables and expert panel review.

GRADE-ADOLOPMENT process for Hyperthyroidism Management EBCPG for Pakistan

GRADEPro evidence to decision Framework

GRADEPro is a web application used to help create, manage, and share summaries of research evidence [ 31 ]. The CCBP staff involved in this study underwent a training module to master use of GRADEPro for the GRADE-ADOLOPMENT process. The software was used to develop Evidence to Decision (ETD) tables to reach a consensus on each of the recommendations marked “ Adapt .”

ETD tables that summarize evidence to enable members of an expert panel to make healthcare recommendations or decisions. Development of ETD tables begins with formulation of a question structured as follows: “Should the Intervention/Suggested Change be favored over the Comparison/Current Standard of Practice? ” The pros and cons of the suggested change are judged by an expert panel across 12 criteria, that are shown in Supplementary Table 1 (Additional file 1 ).

Each criterion was supported with evidence gathered through a best evidence review process (Additional file 1 ), to provide local context for the pros and cons of the recommendation. The CCBP team summarized the newly gathered evidence for each criterion in the “ Research Evidence ” and “ Additional Considerations ” columns. The GRADE-USA working group was deeply involved in the creation of the ETD tables.

Expert panel review

An expert panel of five endocrinology faculty from AKU were invited by the Endocrinology Section Head to review the completed ETD table for each recommendation and provide their judgement for each criterion. This judgment was in the form of a single selection from multiple response options. If, for any criteria, an expert required additional evidence, they informed the CCBP team. An effort was made to source the requisite information, which, if found, was shared with all the panel members. Experts’ judgements were sought in an anonymous and confidential manner, with the GRADEPro software allowing reviewers to select options and provide feedback without their identity known to fellow experts or the CCBP team. A sample of a GRADEPro ETD is shown as Supplementary Table 2 (Additional file 1 ).

Final recommendation revisions & synthesis

Once all the members of the expert panel had provided their responses to the ETD, the CCBP staff synthesized their responses to produce a summary of judgments. The CCBP staff conducted a meeting with the expert panel to review the summary of judgments and reach a final unanimous consensus on the need for and nature of any revisions to the recommendations in question. The strength of each recommendation was also decided. Finally, the consensus was presented to the Section Head of Endocrinology for review, after which the recommendation was incorporated into the Pakistani EBCPG with a summary of the consensus decision.

Final debriefing to identify challenges & explore solutions

Two focus group discussions (FGDs) were conducted to identify challenges faced throughout the entire GRADE-ADOLOPMENT process and to explore corresponding solutions. These FGDs were led by a member of the CCBP team and included the CCBP staff and the Section Head of Endocrinology. Participants were given the opportunity to first brainstorm challenges and solutions independently, and these were then discussed within the FGD. Each challenge was decided as per consensus opinion to be either a major or minor challenge. The CCBP team then categorized the final list of specific challenges within broad themes, and their corresponding solutions were presented alongside them.

Initial review of source guideline

The source guideline included a total of 124 recommendations, out of which 71 were adopted and 49 were excluded. 4 recommendations were carried forward for adaptation via the ETD process (Fig.  2 ) (Supplementary Table 3) (Additional file 1 ). A list of all excluded recommendations can be found in Supplementary Table 4 (Additional file 1 ).

Outcomes of table of recommendations review

Evidence-to-decision (ETD) tables

Amongst the four recommendations that underwent the adaptation process, modifications were made to two (Tables  1 and 2 ), while the remaining two were unchanged (Tables  3 and 4 ). The complete Evidence to Decision tables with the summary of judgements for the modified recommendations can be found in Supplementary Tables 5 & 6 (Additional files 2 & 3 ).

Challenges and solutions

The challenges faced were broadly categorized into four main themes: resources, stakeholder support and involvement, resistance to change, and methodological limitations (Table  5 ). Solutions proposed for the challenges faced will be incorporated in the future updates of the guideline.

In this paper, we applied the GRADE-ADOLOPMENT process to the 2016 American Thyroid Association Guidelines for Diagnosis and Management of Hyperthyroidism and Other Causes of Thyrotoxicosis [ 30 ] to adolop EBCPGs for the management of hyperthyroidism in the local context of Pakistan. Out of a total of 124 recommendations, 71 were adopted, 49 were excluded, and 4 were subjected to the process of adaptation. The adapted recommendations primarily focused on accommodating patient-centered factors and accounting for a lack of resources in Pakistan, without a significant compromise in clinical outcomes.

The first adapted recommendation addressed the need for liver function tests (LFTs) amongst patients experiencing symptoms of hepatotoxicity while being treated with anti-thyroid drugs (ATDs). The source EBCPG recommended a full panel of LFTs (alanine transaminase, aspartate transaminase, alkaline phosphatase, gamma-glutamyl transferase; total, conjugated and unconjugated bilirubin) for all patients experiencing any symptoms remotely suggestive of hepatotoxicity (pruritic rash, jaundice, light-colored stool or dark urine, joint pain, abdominal pain or bloating, anorexia, nausea, or fatigue) [ 30 ]. However, this recommendation was adapted to advise the use of only alanine transaminase (ALT) to diagnose the extent of liver injury in patients experiencing highly specific symptoms (jaundice, pruritis, or change in stool color). The rationale behind this adaptation was centered around the infrequent incidence of hepatotoxicity while on ATD (1.4–6.3% [ 32 ]) and the patient-borne financial ramifications of over-testing. In contrast to high-income countries where government or private insurance covers the majority of healthcare costs, almost 60% of healthcare costs in Pakistan are via out-of-pocket payment by patients [ 33 ], with national health coverage provided to only 20% of the population [ 34 ]. The cost of a full LFT panel in Pakistan ranges from $3.73–7.15, which is between 3 and 7 times more than a single ALT test (ranges from $1.01–1.67). However, while patient finances must be given full consideration in the management of hyperthyroidism, future research is needed to investigate the cost-effectiveness of the adapted recommendation in a Pakistani population.

The second recommendation to undergo the adaptation process was related to thyroid status testing post-treatment by radioactive iodine (RAI) therapy for Graves’ Disease (GD). The source EBCPG recommends assessing free T4 (FT4), total T3, and thyroid-stimulating hormone (TSH) amongst patients within 1–2 months after patients with GD receive RAI therapy, followed by 4–6 weekly testing for 6 months, or until the patient becomes hypothyroid and is stable on thyroid hormone replacement This recommendation was modified to advise the assessment of only FT4 at initial follow-up, with subsequent TSH assessment only in the case of low T4. The keystone of this modification was the consensus that FT4 alone is a sufficiently sensitive modality to detect post-RAI hypothyroidism, and that TSH suppression in the post-RAI period may limit its accuracy in reflecting thyroid status. In fact, this misleading suppression of TSH after RAI therapy may prompt the physician to initiate thioamides unnecessarily. Moreover, in Pakistan, the use of a single FT4 test (ranges from $4.92–8.30) is about a third the price of a full panel consisting of FT4, T3 and TSH ($12.7–18.1). In fact, a sizeable percentage (48.8%) of the overall management costs for hyperthyroidism are attributable to laboratory testing [ 35 ]. Lastly, if both the initial FT4 and subsequent TSH assessment reflect hypothyroidism, and thyroid hormone replacement is initiated and optimized, long-term assessment of treatment effectiveness can be monitored by TSH alone. To facilitate adherence to follow-up and routine post-operative testing, it is recommended that public and private laboratories in Pakistan should partner with healthcare centers to create comprehensive and appropriate care packages which include all post-treatment management and surveillance.

The third recommendation that underwent the adaptation process concerned the preoperative administration of potassium iodide (KI; Lugol’s solution), in addition to ATD and/or beta-blockers, prior to surgical management of GD. While no changes were enacted to this recommendation, experts noted that KI was not widely accessible in Pakistan, with availability of KI being restricted to tertiary care hospitals and large-scale pharmacies, even in urban settings. Though the supporting evidence lacks robustness and clarity, KI is believed to limit intraoperative blood loss by decreasing thyroid vascularity, and also suppress the synthesis and release of thyroid hormone [ 11 ]. However, despite these benefits, the lack of widespread availability of KI in Pakistan would undoubtedly preclude its universal use before surgery for GD. Moreover, recent studies have once again called into question the benefits of preoperative KI administration, with regards to its impacts on intraoperative bleeding, difficulty of operation, operative time, and postoperative outcomes [ 36 , 37 , 38 ]. Thus, the expert team added an additional comment after adopted recommendation, which reassured readers that a lack of administration of KI would likely not compromise the health outcomes of a patient.

The final recommendation that underwent the adaptation process advised a single dose of RAI to render a patient with GD hypothyroid. Although no modifications were made to the recommendation, discussions centered around the cost-effectiveness and availability of RAI versus an alternate option of employing ATD therapy with regular thyroid function test (TFT) monitoring. However, though ATD therapy may provide a more financially feasible mode of treatment, the remission rate of GD with RAI therapy is significantly higher than with ATD therapy [ 35 ]. Therefore, RAI should be considered for definitive treatment in GD patients on high doses of ATD treatment, those not responding to the ATD treatment, and those requiring ATD treatment for more than 2 years.

There are limitations to the GRADE-ADOLOPMENT process used in our study that we would like to acknowledge. Firstly, individual-level (e.g., the Section Head reviewing each ToR to independently to decide whether to adopt, adapt or exclude recommendations) and group-level (e.g., the consensus meeting featuring five experts from a single institution) biases may limit the applicability of our EBCPGs to other settings in Pakistan. Additionally, fundamental to the GRADE-ADOLOPMENT process, the adaptation process was guided primarily by expert consensus, due to the suboptimal availability of local, high-quality level of evidence. Moreover, we limited the inclusion of other important stakeholders, such as patients, allied health professionals, general practitioners, nurses, experts external to AKU, other healthcare centers, external endocrinology organizations or societies, and provincial and federal governments. This was to minimize inevitable delays that would have accompanied a larger team, including logistic difficulties, conflicts of interest, lack of mutual availability, political influences, and lack of direct incentives. However, prior experience in developing such EBCPGs enabled the CCBP team to remain mindful of the needs and values of these groups to a large extent. Lastly, while the efforts to create a local EBCPG for the management of hyperthyroidism have yielded success, the feasibility of widespread utilization and implementation of the EBCPG across Pakistan remains a concern. All the aforementioned limitations represent real-world barriers to the idealistic implementation of the GRADE-ADOLOPMENT process in resource-constrained and poorly structured healthcare systems in LMICs like Pakistan.

The outcome of the GRADE-ADOLOPMENT process applied to the 2016 American Thyroid Association Guidelines for Diagnosis and Management of Hyperthyroidism and Other Causes of Thyrotoxicosis [ 30 ] yielded two major changes in the newly developed Pakistani EBCPG for the management of hyperthyroidism. These included the recommendation to assess only ALT (as opposed to a full LFT panel) amongst patients on ATDS experiencing symptoms highly specific of hepatotoxicity (as opposed to a higher index of suspicion considering non-specific symptoms like bloating, anorexia, nausea, or fatigue), and the recommendation to assess only FT4 (as opposed to the full panel of FT4, total T3, and TSH) at initial follow-up after RAI therapy for GD, with subsequent TSH assessment only in the case of low T4. The rationale behind both these changes were to prioritize patients’ finances during the course of hyperthyroidism management and to limit the overuse of laboratory testing in a resource-constrained setting. Future research must investigate the cost-effectiveness and risk-benefit ratio of these modified recommendations.

Availability of data and materials

All data generated or analysed during this study are included in this published article and its supplementary information files.

Taylor PN, Albrecht D, Scholz A, Gutierrez-Buey G, Lazarus JH, Dayan CM, et al. Global epidemiology of hyperthyroidism and hypothyroidism. Nat Rev Endocrinol. 2018;14(5):301–16.

Article   PubMed   Google Scholar  

Devereaux D, Tewelde SZ. Hyperthyroidism and thyrotoxicosis. Emerg Med Clin. 2014;32(2):277–92.

Article   Google Scholar  

Initiatives MoPDS. Pakistan population situation analysis. Pakistan: UNFPA; 2020.

Akhter S, Khan A, Siddiqui MM, Nawab G. Frequencies of thyroid problems in different age, sex and seasons. J Med Sci. 2001;1:153–6.

Hollowell JG, Staehling NW, Flanders WD, Hannon WH, Gunter EW, Spencer CA, et al. Serum TSH, T(4), and thyroid antibodies in the United States population (1988 to 1994): National Health and Nutrition Examination Survey (NHANES III). J Clin Endocrinol Metab. 2002;87(2):489–99.

Article   CAS   PubMed   Google Scholar  

Garmendia Madariaga A, Santos Palacios S, Guillén-Grima F, Galofré JC. The incidence and prevalence of thyroid dysfunction in Europe: a meta-analysis. J Clin Endocrinol Metab. 2014;99(3):923–31.

Jawa A, Jawad A, Riaz SH, Assir MZ, Chaudhary AW, Zakria M, et al. Turmeric use is associated with reduced goitrogenesis: thyroid disorder prevalence in Pakistan (THYPAK) study. Indian J Endocrinol Metab. 2015;19(3):347–50.

Article   CAS   PubMed   PubMed Central   Google Scholar  

Selmer C, Olesen JB, Hansen ML, von Kappelgaard LM, Madsen JC, Hansen PR, et al. Subclinical and overt thyroid dysfunction and risk of all-cause mortality and cardiovascular events: a large population study. J Clin Endocrinol Metab. 2014;99(7):2372–82.

Haymart MR. The role of clinical guidelines in patient care: thyroid hormone replacement in women of reproductive age. Thyroid. 2010;20(3):301–7.

Article   PubMed   PubMed Central   Google Scholar  

Mestman JH. How effective are clinical guidelines for hypothyroidism in pregnancy in clinical practice? Clin Thyroidol. 2013;25:194–6.

Ross DS, Burch HB, Cooper DS, Greenlee MC, Laurberg P, Maia AL, et al. 2016 American Thyroid Association guidelines for diagnosis and management of hyperthyroidism and other causes of thyrotoxicosis. Thyroid. 2016;26(10):1343–421.

Kahaly GJ, Bartalena L, Hegedüs L, Leenhardt L, Poppe K, Pearce SH. 2018 European thyroid Association Guideline for the management of Graves’ hyperthyroidism. Eur Thyroid J. 2018;7(4):167–86.

Schünemann HJ, Wiercioch W, Brozek J, Etxeandia-Ikobaltzeta I, Mustafa RA, Manja V, et al. GRADE evidence to decision (EtD) frameworks for adoption, adaptation, and de novo development of trustworthy recommendations: GRADE-ADOLOPMENT. J Clin Epidemiol. 2017;81:101–10.

De Leo S, Lee SY, Braverman LE. Hyperthyroidism. Lancet. 2016;388(10047):906–18.

Alam¹ Z, Shah M, Khan M, Ali W, Shehzad A, Shah JA, et al. Thyroid dysfunction and prevalence of both clinical and subclinical form of hyperthyroidism and hypothyroidism in District Mardan, KPK, Pakistan. 2019;8:98–104.

Alvi AM, Azmat U, Shafiq W, Rasheed AHA, Siddiqi AI, Khan S, et al. Efficacy of radioiodine therapy in patients with primary hyperthyroidism: an institutional review from Pakistan. Cureus. 2022;14(5).

Sarwar S. Iodized salt: a risk factor for hyperthyroidism. J Rawalpindi Med Coll. 2013;17(2):284–7.

ADS   Google Scholar  

Olmos RD, De Figueiredo R, Aquino E, Lotufo P, Bensenor IM. Gender, race and socioeconomic influence on diagnosis and treatment of thyroid disorders in the Brazilian Longitudinal Study of Adult Health (ELSA-Brasil). Braz J Med Biol Res. 2015;48:751-8.

Shaffique S, Anwar H, Asif HM, ul Haq I, Akram MJRJP, Sciences P. Prevalence of hyperthyroidism and its impact on Quality of Life among students of the Islamia. Univ Bahawalpur Pakistan. 2020;8(2):85–90.

Google Scholar  

Alonso-Coello P, Oxman AD, Moberg J, Brignardello-Petersen R, Akl EA, Davoli M, et al. GRADE evidence to decision (EtD) frameworks: a systematic and transparent approach to making well informed healthcare choices. Clin Pract Guidelines. 2017;2(2):167.e1-e10.

Okely AD, Ghersi D, Hesketh KD, Santos R, Loughran SP, Cliff DP, et al. A collaborative approach to adopting/adapting guidelines - the Australian 24-hour movement guidelines for the early years (birth to 5 years): an integration of physical activity, sedentary behavior, and sleep. BMC Public Health. 2017;17(Suppl 5):869.

Kahale LA, Ouertatani H, Brahem AB, Grati H, Hamouda MB, Saz-Parkinson Z, et al. Contextual differences considered in the Tunisian ADOLOPMENT of the European guidelines on breast cancer screening. Health Res Policy Syst. 2021;19(1):80.

Darzi A, Harfouche M, Arayssi T, Alemadi S, Alnaqbi KA, Badsha H, et al. Adaptation of the 2015 American College of Rheumatology treatment guideline for rheumatoid arthritis for the Eastern Mediterranean Region: an exemplar of the GRADE Adolopment. Health Qual Life Outcomes. 2017;15(1):183.

Loo BKG, Okely AD, Pulungan A, Jalaludin MY. Asia-Pacific Consensus Statement on integrated 24-hour activity guidelines for children and adolescents. 2022;56(10):539 − 45.

Coronado R, Gómez de León AO, Faba-Beaumont M. Adaptation of clinical practice guidelines for osteoporosis in a Mexican context. Experience using methodologies ADAPTE, GRADE-ADOLOPMENT and RAND/UCLA. J Clin Epidemiol. 2020;131:30–42.

Reilly JJ, Hughes AR, Janssen X, Hesketh KR, Livingstone S, Hill C, et al. GRADE-ADOLOPMENT process to develop 24-hour movement behavior recommendations and physical activity guidelines for the under 5s in the United Kingdom, 2019. J Phys Act Health. 2020;17(1):101–8.

Society PE. Publication & trials. Available from: https://www.pakendosociety.org/publications-trials/ .

Haq IU, Rehman ZU. Medical Research in Pakistan; A Bibliometric Evaluation from 2001 to 2020. Libr Philos Pract. 2021:1–13.

Bhatti MW. Country facing shortage of internal medicine specialists owing to non-practising female doctors. Pakistan: The News International; 2022. thenews.com.pk .

RossDouglas S, BurchHenry B, CooperDavid S, Carol G, Luiza M, RivkeesScott A, et al. 2016 American Thyroid Association guidelines for diagnosis and management of hyperthyroidism and other causes of thyrotoxicosis. Thyroid. 2016;26(10):1343–1421.

GRADEpro G, GRADEpro GDT. GRADEpro Guideline Development Tool [Software]. McMaster University, 2015 (developed by Evidence Prime, Inc.); 2015.

Suzuki N, Noh JY, Hiruma M, Kawaguchi A, Morisaki M, Ohye H, et al. Analysis of antithyroid drug-Induced severe liver Injury in 18,558 newly diagnosed patients with Graves’ disease in Japan. Thyroid. 2019;29(10):1390–8.

Statistics PBo. Government of Pakistan Bureau of Statistics Karachi. 2021. Available from: https://www.pbs.gov.pk .

Cheema AR, Zaidi S, Najmi R, Khan FA, Kori SA, Shah NA. Availability does not mean utilisation: analysis of a large micro health insurance programme in Pakistan. Global J Health Sci. 2020;12(10):1–4.

Zuberi LM, Awan S, Islam N, Akhter J, Jabbar A. Does choice of therapy save costs and improve outcomes in hyperthyroid patients. J Pak Med Assoc. 2008;58(6):309.

PubMed   Google Scholar  

Lindner K, Kußmann J, Fendrich V. Preoperative potassium iodide treatment in patients undergoing thyroidectomy for Graves’ disease—perspective of a European high-volume center. World J Surg. 2020;44(10):3405–9.

Randle RW, Bates MF, Long KL, Pitt SC, Schneider DF, Sippel RS. Impact of potassium iodide on thyroidectomy for Graves’ disease: implications for safety and operative difficulty. Surgery. 2018;163(1):68–72.

Kalra S, Kalra B. Mishti copes with diabetes: a pragmatic approach to coping skills training. J Soc Health Diabetes; New York. 2015;5(1):1–2.

Download references

Acknowledgements

Author information.

Russell Seth Martins and Sarah Nadeem have contributed equally to this manuscript and wish to be considered as joint first authors.

Authors and Affiliations

Center for Clinical Best Practices, Clinical and Translational Research Incubator (CITRIC), Aga Khan University, Karachi, 74800, Pakistan

Russell Seth Martins, Sarah Nadeem, Abeer Aziz, Alina Pervez, Nashia Ali Rizvi & Mohsin Ali Mustafa

Section of Endocrinology, Department of Medicine, Aga Khan University, Karachi, 74800, Pakistan

Sarah Nadeem, Najmul Islam, Asma Ahmed, Aisha Sheikh, Saira Furqan, Nanik Ram, Azra Rizwan & Muhammad Qamar Masood

FACE (Fellow American College of Endocrinology), Internal Medicine & Endocrinology, Diabetes & Metabolism, Internal Medicine, and Endocrinology, Women in Medicine Committee, Associate Dean’s Women Faculty Forum, Aga Khan University, Karachi, Pakistan

Sarah Nadeem

Medical College, Aga Khan University, Karachi, 74800, Pakistan

Sajjan Raja

Department of Medicine, Aga Khan University, Karachi, 74800, Pakistan

Salima Saleem Aamdani

Learning Research Centre, Patel Hospital, Karachi, 75300, Pakistan

Bushra Ayub

You can also search for this author in PubMed   Google Scholar

Contributions

RSM, NI, AAh, AS, SF, NR, AR, NAR, MAM, SSA, BA, SN and MQM were involved in the conceptualization of the manuscript. NI, AAh, AS, SF, NR, AR, NAR, MAM, SSA, BA, SN and MQM were involved in the GRADE-ADOLOPMENT of the EBCPGs. RSM, AAz, SR, AP, and SN were involved in the writing of the manuscript. The final draft was reviewed by all authors.

Corresponding author

Correspondence to Sarah Nadeem .

Ethics declarations

Ethics approval and consent to participate.

Given the lack of involvement of patients or other human participants, a waiver of ethics approval and informed consent was obtained from the Ethics Review Committee of the Aga Khan University. All methods were conducted in accordance with the highest ethical standards outlined in the 1964 Declaration of Helsinki and its future amendments.

Consent for publication

Not applicable.

Competing interests

The authors declare no competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Additional file 1., additional file 2., additional file 3., rights and permissions.

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Martins, R.S., Nadeem, S., Aziz, A. et al. GRADE-ADOLOPMENT of hyperthyroidism treatment guidelines for a Pakistani context. BMC Endocr Disord 24 , 41 (2024). https://doi.org/10.1186/s12902-023-01493-1

Download citation

Received : 31 October 2022

Accepted : 17 October 2023

Published : 21 March 2024

DOI : https://doi.org/10.1186/s12902-023-01493-1

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

BMC Endocrine Disorders

ISSN: 1472-6823

• General enquiries: [email protected]

An official website of the United States government

The .gov means it's official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you're on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings
• Browse Titles

NCBI Bookshelf. A service of the National Library of Medicine, National Institutes of Health.

Viner RM, Kinra S, Christie D, et al. Improving the assessment and management of obesity in UK children and adolescents: the PROMISE research programme including a RCT. Southampton (UK): NIHR Journals Library; 2020 Mar. (Programme Grants for Applied Research, No. 8.3.)

Improving the assessment and management of obesity in UK children and adolescents: the PROMISE research programme including a RCT.

Chapter 9 future research recommendations.

Recommendations for future research directions are made in each chapter and substudy, and are summarised here.

• Study A: scoping the impact of the National Child Measurement Programme on the childhood obesity pathway

It was found that NCMP feedback has a role in changing parental perceptions and health knowledge in the general population, but improvements or other initiatives may be required to precipitate lifestyle behaviour changes that will reduce childhood obesity prevalence.

First, in terms of methods, future research should seek to evaluate behaviour change and BMI change in the NCMP more accurately, for example through assessment and measurement of children before and after feedback. This would allow more accurate assessment of the ability of the NCMP to precipitate change.

Second, further work is needed to understand how best to address the marked and persistent discrepancy found between NCMP-reported weight status and parental perceptions. This appeared to have a key mediating role in whether or not parents made changes as a result of NCMP feedback. This work should include a better understanding of the causes of inaccurate perception and development of potential strategies to redress this. Consideration of inequalities will be essential in this, given the findings that deprivation and ethnicity were strongly associated with accuracy of weight perception, and also with likelihood to make changes after feedback.

Third, given the findings that parents report inconsistent and sometimes contradictory advice from health professionals regarding NCMP feedback, work is needed to understand primary care health professionals’ awareness and understanding of NCMP feedback and to understand how they deal with NCMP feedback in managing childhood obesity. Both quantitative and qualitative work are needed to identify barriers for health professionals to helping parents respond to NCMP feedback and to help professionals initiate difficult conversations about weight with children.

Finally, there is wide scope for investigation of potential improvements to delivery of NCMP feedback to parents. This clearly depends on collaboration with local authorities and PHE. The findings suggest a number of potential further avenues to explore in terms of modifying feedback to parents. Key among these would be investigation of how feedback could be more motivational and whether or not methods of feedback have utility in improving parental awareness of overweight and acceptance of NCMP findings, for example use of more graded systems for representing weight status feedback.

• Study B: developing a new electronic tool to improve childhood obesity management in primary care

The key lines for further research in this study relate to further development and evaluation of the tool. The first stages in the recognised pathway for development of a complex intervention were undertaken, namely modelling, development and a feasibility pilot study. A larger pilot was aimed for; however, this did not prove feasible.

The logical next steps are a randomised pilot study of CATCH and then, potentially, an efficacy trial. The randomised pilot should be in a different and broader geographical area than the small number of practices in London used in this study. To reduce time and redundancy, the pilot could be an internal pilot with direct relation to the larger trial.

However, further developmental work of the tool is needed as well. It is believed that further work should be done to improve the accuracy of the risk assessment tool by updating risk assessment analyses using additional cohorts, if data are available. Second, there should be consideration of inclusion of training for professionals in the tool, given that practitioners feeling uncomfortable discussing weight-related health problems with parents was found to be a concern.

• Study C: the Healthy Eating and Lifestyle Programme randomised clinical trial

It is unlikely that there is further work to be done on the model of the HELP intervention delivered in this trial, given the negative findings. However, given that HELP was developed as a intervention delivered by clinical psychologists, and that there were promising pilot data relating to its use in that setting, it would be useful to further evaluate the delivery of HELP in a more clinical setting with more expert providers. This evaluation should take the form of a randomised pilot, potentially followed by an efficacy trial.

Given the negative findings in this study, and similar negative findings by many other programmes, as noted in Chapter 4 , Discussion , further work is needed to understand how weight management programmes can be delivered effectively to young people from diverse and often deprived backgrounds in which childhood obesity is common. This should consider the barriers to and facilitators of effective lifestyle weight management in deprived children and young people. Furthermore, given the finding of very high levels of psychological distress and disorder in this trial population, work should be undertaken to examine how standard lifestyle modification weight management can be tailored and delivered effectively to young people with significant psychological distress or disorder.

• Study D: evaluation of anti-obesity drug treatment in children and adolescents

Study D findings highlighted a number of directions for future research. Although AOD treatment options are currently limited for young people (and adults), future new drugs are likely to share many of the characteristics and issues found with current drugs.

In terms of the findings that use of, persistence with and confidence in prescribing AODs were each poor to very poor, future research needs to consider how to improve the prescribing of and experience of AODs for young people. First, studies should consider combining AOD use with formal behavioural lifestyle modification programmes. There is some suggestion from the adult literature that such combinations are likely to be most effective, but such trials have not been carried out in adolescents. Second, for orlistat, the findings suggest that some of the reasons for cessation related to poor preparation of patients (in terms of awareness of effects and side effects) and training of prescribing teams (in preparing patients). Work should examine how both of these barriers can be overcome, with support for patients being provided within a behavioural framework, as outlined above. Furthermore, for orlistat, consideration should be given to study of its utility as a diet training aid, that is in providing feedback on levels of fat in the diet, rather than as a weight loss drug per se. Development and evaluation of training for health professionals should be undertaken to improve utility of AODs, both for current and future generations of drugs.

In terms of the evidence base for AODs in adolescence, this is largely limited to short-term use; there is a need for long-term data on AOD interventions in adolescence, in terms of both efficacy and adverse events. AOD trial populations also need to be broadened to young people with specific comorbidities, for example those with learning disabilities, those with psychological disorders and those with T2DM, as these are currently excluded from AOD trials, yet are groups with a high obesity burden.

• Study E: evaluation of acceptability and early outcomes of adolescent bariatric surgery in the UK

The range of studies on the very new area of adolescent bariatric surgery highlighted a range of future avenues for research, perhaps not surprising given the newness of the field and the paucity of existing research.

In terms of surgery itself, future work should move past individual hospital series and collect comparable consistent data across all centres nationally. These data should include routine collection of psychological, health economic and QoL data, to enable routine monitoring of outcomes over the shorter and longer term. The need for longer-term data is particularly acute. Psychological and QoL data are necessary to understand which young people benefit most from surgery and, thus, potentially allow clinicians to make more informed decisions about which young people are most likely to benefit from surgery. Routine collection of data on metabolic comorbidities and bone density are also necessary to inform studies of the long-term impacts of surgery in young people.

Further work is needed to help clinicians move towards a shared decision-making model with young people and families. This could take the form of an implementation science study building on the findings of the present decision-making study. As part of this, it would be important to investigate and evaluate mechanisms to reduce dropout from follow-up after surgery and in long-term follow-up, and whether or not this dropout is related to involvement in decision-making pre surgery.

Other work should examine whether or not the evidence of cost-effectiveness identified here for adolescent bariatric surgery within the NHS is also found in other health systems.

• Cite this Page Viner RM, Kinra S, Christie D, et al. Improving the assessment and management of obesity in UK children and adolescents: the PROMISE research programme including a RCT. Southampton (UK): NIHR Journals Library; 2020 Mar. (Programme Grants for Applied Research, No. 8.3.) Chapter 9, Future research recommendations.
• PDF version of this title (11M)

In this Page

Other titles in this collection.

• Programme Grants for Applied Research

Recent Activity

• Future research recommendations - Improving the assessment and management of obe... Future research recommendations - Improving the assessment and management of obesity in UK children and adolescents: the PROMISE research programme including a RCT

Your browsing activity is empty.

Activity recording is turned off.

Turn recording back on

Connect with NLM

National Library of Medicine 8600 Rockville Pike Bethesda, MD 20894

Web Policies FOIA HHS Vulnerability Disclosure

Help Accessibility Careers

Developing a Framework for Self-regulatory Governance in Healthcare AI Research: Insights from South Korea

• Perspective
• Open access
• Published: 25 March 2024

Cite this article

You have full access to this open access article

• Junhewk Kim   ORCID: orcid.org/0000-0002-9109-270X 1 ,
• So Yoon Kim   ORCID: orcid.org/0000-0001-7015-357X 2 ,
• Eun-Ae Kim   ORCID: orcid.org/0000-0002-6989-559X 3 ,
• Jin-Ah Sim   ORCID: orcid.org/0000-0002-3494-3002 4 ,
• Yuri Lee   ORCID: orcid.org/0000-0003-0584-650X 5 &
• Hannah Kim   ORCID: orcid.org/0000-0003-2938-9745 2  

This paper elucidates and rationalizes the ethical governance system for healthcare AI research, as outlined in the ‘Research Ethics Guidelines for AI Researchers in Healthcare’ published by the South Korean government in August 2023. In developing the guidelines, a four-phase clinical trial process was expanded to six stages for healthcare AI research: preliminary ethics review (stage 1); creating datasets (stage 2); model development (stage 3); training, validation, and evaluation (stage 4); application (stage 5); and post-deployment monitoring (stage 6). Researchers identified similarities between clinical trials and healthcare AI research, particularly in research subjects, management and regulations, and application of research results. In the step-by-step articulation of ethical requirements, this similarity benefits from a reliable and flexible use of existing research ethics governance resources, research management, and regulatory functions. In contrast to clinical trials, this procedural approach to healthcare AI research governance effectively highlights the distinct characteristics of healthcare AI research in research and development process, evaluation of results, and modifiability of findings. The model exhibits limitations, primarily in its reliance on self-regulation and lack of clear delineation of responsibilities. While formulated through multidisciplinary deliberations, its application in the research field remains untested. To overcome the limitations, the researchers’ ongoing efforts for educating AI researchers and public and the revision of the guidelines are expected to contribute to establish an ethical research governance framework for healthcare AI research in the South Korean context in the future.

Avoid common mistakes on your manuscript.

Introduction

The rapid progress of machine learning and artificial intelligence (AI) poses new and unprecedented challenges to the entire healthcare sector. Particularly, as a critical extension of the foundational discussions on the technology adoption in healthcare (Rajpurkjar et al. 2022 ), the focus now shifts towards the practical governance and regulation of AI development and its application in healthcare landscape. South Korea has swiftly embraced biomedical technologies, showcasing a clear inclination in integrating AI in healthcare. The ‘2022 Medical Device License Report’ from the Ministry of Food and Drug Safety (MFDS) of the Republic of Korea unveils that a total of 149 AI-based medical devices obtained approval and certification in the country, with 10 receiving approval and 38 attaining certification in 2022 (MFDS 2023 ).

Corresponding with this trend, the Korean National Institutes of Health (KNIH) published the ‘Research Ethics Guidelines for AI Researchers in Healthcare’ in August 2023, marking an initial effort to offer an actionable guidance to healthcare AI researchers in the country (KNIH 2023 ). The guidelines aim to establish ethical standards for all stages of healthcare AI development by presenting ethical principles and detailed values. The researchers mainly participated in developing the guidelines using robust research methodologies, such as literature reviews, interdisciplinary consultations, and a public hearing as well as providing empirical research evidence from surveys for the lay public and experts. Consequently, the guidelines present six principles with corresponding codes and explanations. The principles, stemmed from the World Health Organization (WHO) report ‘Ethics and governance of artificial intelligence for health’, are tailored for the national context, providing a framework for researchers to evaluate their research practices. Importantly, it is noted that while bioscientists are well-versed in the ethical procedures and legal regulations related to human subjects research, those in computer science and data science engaged in healthcare AI research may lack familiarity with these standards (Metcalf and Crawford 2016 ; Throne 2022 ). Consequently, these guidelines are designed to assist support healthcare AI researchers in conducting ethical research by presenting providing part I principles to consider in relation to research, part II corresponding relevant research codes, regulations and related ethical cases, and part III an expanded framework, aligning with that applies for the existing governance framework for phase I–IV clinical research, tailored for to the context of healthcare AI research.

The purpose of this paper is to outline and provide rationales for the ethical governance system introduced in the part III of the guidelines. At present, we are in the process of translating the guidelines for an official English version. Amidst this ongoing endeavour, this paper preliminarily introduces the final section of the guidelines, which is under linguistic review. Subsequently, we describe the governance framework, comprising six steps, accompanied by ethical and institutional explanations for each stage. In conclusion, this paper presents a healthcare AI research governance system, expanding upon the existing human subjects research. It advocates for the establishment of a robust, secure, and sustainable research governance structure by adapting the clinical research system prevalent in countries where such approaches are already established to the domain of healthcare AI research governance.

Procedural Considerations for Conducting Healthcare AI Research

Aforementioned, part I of the guidelines provide background, developing process and methodologies, aims, scope, and key terms. Next, part II reviews the existing legal frameworks related to safety and effectiveness, liability for errors and negligence, privacy laws for patient data protection, and legal frameworks responding to bias and discrimination. Based on the legal background, part III introduces a six principle-based framework and explanations with specific ethical cases, aligning with the procedural considerations when researchers conduct healthcare AI research.

Particularly, part III of the guidelines is grounded in six ethical principles: (a) respect for and protection of human autonomy; (b) promotion of human well-being, safety, and the public interest; (c) ensuring transparency, explainability, and reliability; (d) upholding accountability and legal obligations; (e) promoting inclusivity and equity; and (f) fostering responsiveness and sustainability (Kim et al. 2023 ). While these principles align with those of the WHO, specific codes and applications have been tailored to suit the national context. The healthcare AI research governance framework presented herein also follows this approach, incorporating relevant principles to be considered at each stage.

The guidelines restructured the principles by the steps of the research process as a form of a checklist. This checklist provides a baseline for all stakeholders the field to voluntarily identify and assess the ethical considerations pertinent to practical research and development (Table  1 ).

Healthcare AI research and development begins with the establishment of a robust ethical framework, grounded in the aforementioned six ethical principles. A multidisciplinary team collaborated to establish the ethical considerations for research and development and delineate the requisite compliance measures. The AI development process comprises distinct stages: data collection, algorithm development, model training integration, and evaluation. Each steps follows a structured ethical framework, integrating the principles, thereby ensuring the ethical integrity of healthcare AI research and development. Periodic evaluations are conducted to assess ethical compliance and identify areas for improvement. Furthermore, continuous feedback is sought following the application of the developed model in real-world environment.

For research institutions, the guidelines play a pivotal role in ensuring ethical standards of healthcare AI research and development. The research institutions can utilize the guidelines to evaluate the design procedure, algorithm development, and application of AI technologies in their own research endeavours. This assessment entails evaluating the alignment of the guidelines with domestic laws, international norms, and societal dialogues. Additionally, it is advisable for review committees and institutions that oversights healthcare AI research and development to implement reasonable and responsible regulations to manage research activities, educating and informing stakeholders about these regulations, and maintaining open communication for ongoing revisions and amendments as required.

Furthermore, through such feedback and societal discussion, the developers of this guidelines strive for continuous refinement, aiming to foster a research environment that esteems ethical principles and values.

Stage 1. Preliminary Ethics Review

Prior to the commencement of healthcare AI research and development, it is imperative to establish a clear ethical framework guided by specific guidelines. This preliminary stage is the responsibility of the organization, tasked with laying the foundational groundwork. They should actively seek advice through public participation action from a diverse array of stakeholders, including patients, the public, and expert groups such as medical ethicists and legal scholars, to ensure a well-rounded perspective through public participation action. Additionally, it is essential to establish and consider ethical guidelines that are particularly relevant to the research and development process, setting a strong foundation for responsible and ethical AI innovation in healthcare.

Related questions:

Does the plan include sensitive objectives? Is the objective to develop a medical device or other health and public health objectives? (Specify clinical diagnosis-treatment decision, patient decision support, prevention, behavioural intervention, public health, and if others, additional descriptions should be included in the protocol.)

Is it human subject research or research utilizing datasets? (check bioethics exemptions and compliance requirements.) If human subjects research, does the plan include interventions or interactions?

Does the plan address potential or manifest harms? (Provide a risk-benefit analysis.)

Is there evidence or potential for sample bias in the plan?

Stage 2. Creating Datasets

In the process of collecting and processing data for healthcare AI model development, several key considerations must be addressed. Initially, it is essential to evaluate the collectability, availability, and intended use of the data. Depending on the potential risk for privacy infringement, appropriate measures such as anonymization or pseudonymization should be employed for the dataset. A detailed data collection plan is crucial to outline the methods and objectives clearly. Additionally, conducting ongoing quality control is imperative to minimize data bias and ensure the diversity and representativeness of the datasets, which are fundamental for the development of fair and effective healthcare AI systems.

Is the data collection plan comprehensive? (identification and consultation with data subjects or maintaining organizations, data types and details, collection techniques, frequency selection, inclusion and appropriateness of purposes of use)

Are anonymization measures considered? (detailed technical and administrative/physical measures; if not anonymized, justification and additional measures required)

Is the dataset size aligned with the learning task and model complexity?

Is the data quality recognized as high?

Are the data appropriately visualized and exploratory analyses conducted?

Is the raw data collected according to approved clinical standards and protocols, utilizing valid and reliable techniques?

Are regular and continuous data quality control measures implemented?

Stage 3. Model Development

Configuring algorithms to align with research objectives and applying preliminary data to assess appropriateness is a critical phase in AI development. Developers should build the model using decision-making algorithms aimed at achieving specific, predefined goals. To ensure transparency, a concise description of the development plan should be publicized, detailing the steps and intentions behind the model’s construction. Standardizing the data before training the model is essential to ensure consistency and accuracy. Additionally, it is crucial to specify any methodological considerations that might reveal bias within the dataset, thereby allowing for adjustments and improvements to maintain integrity and fairness in the model's outcomes.

Related question (considerations in Stage 1 should be considered in conjunction with those below)

Does the plan provide an adequate accounting of human subjects and data subjects?

Are the methods of split cross-validation of datasets and datasets utilized in the plan appropriate? (correcting erroneous data, resolving inconsistencies in data, deleting unnecessary data, ensuring quality assurance and accuracy of data)

Are potential issues with privacy addressed? (review for possible data breach)

Does the plan assess the sources or likelihood of sampling/evaluation/algorithmic bias? (considering resampling, algorithmic fairness, etc.)

Stage 4. Training, Validation, and Evaluation

The phase of training and validating algorithms using the collected data, followed by an evaluation of their applicability for research purposes, is crucial for crafting robust AI systems. Training AI models meticulously is fundamental to boost their reliability and accuracy. It is also critical to ensure that the AI models undergo thorough internal validation through appropriate procedures to confirm its effectiveness and safety in practical applications. Moreover, implementing measures to assess clinical reliability is necessary for healthcare AI development. This includes evaluating the AI’s accuracy, its relevance to clinical applications, the fairness of its decision-making processes, and the level of trust or acceptance these systems receive from both patients and healthcare professionals.

Does the model use a transparent methodology for AI data mining and project implementation? (e.g., CRISP-DM, Footnote 1 KDD, Footnote 2 SEMMA, Footnote 3 CPMAI Footnote 4 )

What is the model’s purpose? (specify predictive models, text mining, automation, record abstraction, biometrics, and if others, additional descriptions should be in the protocol)

What kind of technology is utilized? (specify machine learning, deep learning, natural language processing, unsupervised learning, reinforcement learning, and if others, additional descriptions should be included in the protocol.)

Can any unexpected results be analysed or tracked?

Stage 5. Application

Ensuring compliance with ethical frameworks and legal regulations is paramount when governing AI models in the real-world application. AI models functioning as medical devices, tasked with analysing data for disease diagnosis, management, and prediction, must comply with approval and review protocols established by relevant regulatory bodies. Those covered by health insurance require safety, effectiveness, and economic evaluations by designated authorities. Implementing an external validation process that involves public participation can further reinforce the model’s integrity and social acceptance.

Furthermore, it becomes crucial that clinical AI algorithms to prioritize user-friendliness, requiring minimal training to lessen cognitive load and streamline decision-making. Supervising and maintaining the models involve assessing their ethical integrity and making continuous improvements as necessary. Clearly designate a specific individual or entity responsible for the ethical management of the model.

Is there a match between the dataset and the population setting for model application?

Are the results interpretable?

Have they been assessed for major biases? (e.g., gender, race)

Has the model been externally validated using datasets from other settings?

Has the model been empirically evaluated for validity, clinical utility, and cost-effectiveness?

Stage 6. Post-deployment Monitoring

Continuing engagement with model users and refining the model based on their feedback is essential in this stage. It involves regularly reviewing the model’s performance in real-world applications, aligning with the self-constructed ethical framework previously established. Maintaining open communication and collaboration with all stakeholders, including AI providers, users, patients, the public, and government agencies, is crucial for ongoing development and alignment with user needs and ethical standards. Furthermore, ensuring that the models can be seamlessly integrated into existing production environments is vital for effective decision-making based on real data. This stage emphasizes the importance of adaptability and responsiveness to the evolving landscape of AI applications and societal impacts.

Do you regularly monitor the product whether the entire data process is correctly aligned or when the entire process is performed automatically without the need for human intervention?

Does the user (healthcare provider), user organization (healthcare organization) regularly disclose usage results, both positive and negative?

Are there communication and recovery protocols established for model application errors?

Are there improvements needed in the relevant ethical framework and guidelines?

A Step-by-Step Explanation of Healthcare AI Research Governance Framework

The healthcare AI research governance framework delineated above adapts and extends the phase I–IV process for human clinical research to healthcare AI research. This adaptation allows guideline developers to manage and regulate research more reliably by extending existing research governance procedures, thus reducing the need for designing new schema for healthcare AI research ethics. This approach reduces training efforts and provides a foundation for researchers to quickly comprehend and apply the governance framework. Additionally, many of the administrative resources already established for human subjects research can be leveraged for healthcare AI research.

However, it is imperative to analyse the commonalities and divergences between clinical trials and healthcare AI research. This paper presents the similarities in terms of (a) research subjects, (b) areas of research management and regulation, and (c) application of research results. On the other hand, there are differences between clinical trials and healthcare AI research, including (a) the research and development process, (b) evaluation of research results, and (c) the modifiability of research results.

Firstly, human subjects, biospecimens, or populations in clinical trials share qualitative similarity with health data, their constructs, or databases utilized in healthcare AI research. For instance, biospecimens are recognized for their uniqueness—characteristics derived from the individuals they originate from—and then, health data collected from human subjects possess the same ontological nature as derivatives of individuals. They inherently refer to persons and are intricately connected to them (Cha and Kim 2022 ). Health datasets encapsulate various biological, behavioural, and socioeconomic records of a specific data subject, directly linked with the human body. The linkage of whole genome sequencing (WGS) data to personal identity intertwines the human body with the data presenting (Li et al. 2014 ). In population studies, the population database reflects the target population group, and eventually, they should become ontologically and practically identical.

Secondly, both clinical trials and healthcare AI research aim to derive results that benefit humans—whether it is treatments, new drugs, medical technologies, and biomaterials in clinical trials, or algorithms and applications in healthcare AI research. Just as clinical research with human subjects has established protocols to ensure respect and protection of individuals involved and affected by research process and its outcome (National Commission for the Protection of Human Subjects of Biomedical & Behavioral Research 1978 ), healthcare AI research also confronts to address ethical considerations arising from both the research process and the utilization of its outcomes. The considerations encompass aspects ranging from the respecting and protection of individuals to issue of accountability and sustainability. Similar to the human subjects research oversight by Institutional Review Boards (IRBs), which review and monitor all biomedical research, healthcare AI research necessitates a robust review and monitoring process. This process is crucial even when certain research activities might be exempt from regulatory requirements, acknowledging the unique challenges and potential risks associated with AI. A tailored oversight mechanism for healthcare AI is imperative that all research involving human subjects—or their data—is conducted responsibly and ethically. As human clinical trials aim to apply developed treatments and new drugs to humans by assessing efficacy and safety, healthcare AI research endeavours to apply developed algorithms and applications to humans to demonstrate effectiveness.

Recognizing the identified similarities, it could be argued that the governance framework established for human clinical research can be directly applied to healthcare AI research. However, significant differences between human clinical research and healthcare AI research necessitate a tailored approach.

Primarily, a distinction lies in the development process between human clinical research and healthcare AI research. Human clinical research focuses on developing of treatments or new drugs, validated through assessments of safety and effectiveness and comparative benefit analyses. Upon affirming these steps, a treatment or drug is considered developed, thereafter maintained through post-marketing/application monitoring or management. Conversely, healthcare AI research entails an iterative process of development, refinement, and validation of algorithms or applications, inherently characterized by their modifiability (Higgins and Madai 2020 ). This research paradigm encompasses a series of stages from data collection to algorithm application and continual revision through feedback loops. Throughout the progress, algorithms are expected to continuously learn, revise, and evolve (Pianykh et al. 2020 ). Therefore, a governance approach tailored to this process, spanning from data collection and algorithm development to model training integration, and evaluation becomes essential.

The primary difference consequently leads to variations in how research outcomes are evaluated and modified. Clinical trials typically employ statistical validation methods like randomized controlled trials (RCTs) or equivalent methodologies to confirm effectiveness. In contrast, healthcare AI research assesses performance using metrics such as the area under the receiver operating characteristic (ROC) curve (AUC) derived from collected data (Wu et al. 2021 ), which involves trade-offs between false positives and false negatives. In addition, drugs and medical devices approved through clinical trials are subject to re-evaluation if modifications are made. However, in healthcare AI research, accepting modifications poses a challenge due to its continuous learning nature, disrupting the notion of a consistent “product-based view” (Gerke et al. 2020 ). Therefore, it is practical for governing healthcare AI research governance to consider adopting elements maintainable from the human subjects clinical research governance system while modifying them to suit the development and application dynamics of healthcare AI.

Six-Stage Process for Healthcare AI Research

Given these considerations, this guidance extends the traditional four-phase clinical research process (phase I: safety; phase II: efficacy and side-effects; phase III: large trials; phase IV: post-market surveillance) by introducing a six-stage process for healthcare AI research. The introduction of <Stage 1: preliminary ethics review > and < Stage 2: creating datasets > reflects the unique nature of healthcare AI research and emphasizes the necessity for comprehensive and sustainable research guidelines from data collection stage onwards. < Stage 3: model development > , < Stage 4: training, validation, and evaluation > , < Stage 5: application > , and < Stage 6: post-deployment monitoring > align with the concepts of phases I–IV of clinical research but are specifically tailored to address the characterized process of developing and applying healthcare AI algorithms.

Stage 1 necessitates researchers and developers to establish an ethical framework tailored to their research objectives. This endeavour enables the research organizations and their members to review and establish their own ethical frameworks and establish and operate a framework that is appropriate for their research purposes. Given the diverse nature of healthcare AI, the selection and explicit delineation of an appropriate ethical framework are crucial. The first stage supports engagement of a diverse array of experts and the public, including ethicists, legal scholars, patients, and laypersons to take an interest in the AI research process as necessary. Their collective input serves to establish guiding principles and rules crucial for the ethical conduct of research. This proactive approach aims to promote self-regulated ethical practices among researchers, distinct from mere compliance with legal regulations. Notably, the established ethical framework in stage 1 should be consistently referenced in most subsequent documentation.

Stage 2 specifies plans for data collection and processing, mandating the creation of suitable datasets by designated data creator or “data curators” responsible for assembling and maintaining datasets (Leonelli 2016 ). The data collection and processing activities of researchers undergo to review by the Data Review Boards (DRBs). This board, established to oversight the ethical conduct of data-related procedures, evaluated data collection plan, anonymization methods, dataset size, quality, and management. The DRB operates within the research institution or as an independent body. Proposed by the Ministry of Health and Welfare of South Korea in the “Guidelines for Utilization of Healthcare Data,” the DRBs function as a committee of five or more individuals. Its responsibilities include assessing the suitability of processing pseudonymized information within an institution, reviewing the adequacy of pseudonymization, and managing the use of pseudonymized information within and outside the institution (Ministry of Health and Welfare of South Korea Dec 2022 ). This paper proposes that the DRB or a data appropriateness review entity comprising researchers, developers, and external members. This entity would review the data collection and management system before commencing healthcare AI research. Such proactive review aims to ensure the safety, appropriateness, feasibility, and absence of biases in data utilization for healthcare AI research.

Stage 3 involves the selection and preliminary assessment of algorithms, making the initiation of full-scale research. At this stage, researchers and developers undergo an IRB review encompassing all facets of conducting the study. They are required to provide extensive justifications concerning the study’s objectives, data standardization, and potential biases. The IRB, compared to the DRB, evaluates the appropriateness of the algorithm, the predictability and validity of results based on initial dataset, the reliability and safety of data management, and ensures the unbiased use of algorithm and data. Researchers, for reporting their conduct to the IRB, should consistently refer to the ethical framework established in stage 1. Considering that data utilization might vary concerning the algorithm used, distinct review rules are set by the DRBs and the IRBs. The former focuses on data management practice, while the latter oversees data utilization practices. This stage functions similar to phase I where the accuracy and appropriateness of the algorithm are determined and reviewed based on validated preliminary data. It can be paralleled with phase I safety assessments in clinical trials, wherein the interaction of an experimental medical device or drug with the human body is examined based on a small number of research subjects.

Stage 4 encompasses the training, validation, and evaluation of the algorithm using the collected real-world data. The training data should be divided into train and test sets, and a pre-prepared validation set, distinct from the training data, is essential for validating healthcare AI algorithms to prevent overfitting and assess real-world applicability. The management of the validation process is imperative to avoid the exportation of models that are only useful during the training process to the actual application phase. and it is recommended that the research and development organization check this process. In the context of healthcare AI applications such as diagnostic imaging, patient risk prediction, and personalized treatment planning, each employing base algorithms ranging from deep learning to decision trees, the need for tailored validation processes becomes clear. For diagnostic imaging or patient risk prediction models, the validation process should primarily focus on rigorous statistical evaluation to ensure accuracy and reliability. Personalized treatment planning systems necessitate validation that emphasizes clinical relevance and the improvement of patient outcomes. These validation processes are essential for assessing the reliability of healthcare AI models. This stage can be seen as akin to phase II in clinical research, the phase that evaluates the effectiveness of a medical device or drug against a placebo. The emphasis is particularly placed on validating the trained algorithm and its relevance to clinical procedures.

Stage 5 involves the deployment of the developed healthcare AI algorithm into practical settings. The regulatory landscape governing healthcare AI implementation may vary based on its real-world application within a country. In South Korea, for example, AI model is evaluated and approved as a medical device by the Ministry of Food and Drug Safety. Moreover, for seeking for the National Health Insurance reimbursement, assessing safety, effectiveness, and economic evaluation from responsible regulators are mandatory. Throughout the step, the organization requires to pursue external validation for its development process, algorithms, and applications while prioritizing transparency. Furthermore, since the nature of healthcare AI includes continuous learning and development as part of its attributes, stage 5 also assigns responsibility for ongoing monitoring, identifying the entity accountable for managing the model. This stage corresponds to phase III, large trials, in clinical trials, where large-scale RCTs are used to determine the applicability of a treatment or new drug, in terms of determining the real-world applicability of a healthcare AI algorithm and putting it to work in the field.

Stage 6 mandates all parties involved to review the process of the continued deployment and ongoing development once the developed algorithm or model has been put into operation in a healthcare setting. Continuous review of use of the model and the functionality of the ethical framework remains pivotal. Maintaining transparent and collaborative communication among all stakeholders emerges as a necessity. In addition, vigilant monitoring of ongoing evolution of the model is imperative to prevent that decision-making based on real-world data might lead to unintended harms. This phase emphasized the follow-up and surveillance of algorithms and models post-launch, analogous to phase IV, post-market Surveillance in clinical research, which refers to the follow-up phase after clinical implementation of a medical device or drug.

The six-stage healthcare AI research governance proposed in this study can be compared to the five-phase standard, BS30440, recently proposed by the UK (Sujan et al. 2023 ). Set to take effect in the second quarter of 2023, BS30440 provides guidelines for validating AI systems in healthcare in the UK context. The guidelines reflect the product life-cycle of healthcare AI, which consists of inception, development, validation, deployment, and monitoring. Compared to the UK guidelines, the six stages presented in this paper add a preliminary ethical framework design and committee verification of data collection and management, distinguishing stages between algorithm determination and subsequent training, validation, and evaluation. BS30440 lacks stipulations for preliminary procedures or data management, integrates algorithm determination and training as a singular process, and makes validation as a separate process. Notably, our study’s governance procedure is designed to extend existing clinical research management procedures, whereas BS30440 establishes novel procedures. This study only examines these distinctions not to favour one framework over the other but to underscore the global development and application of similar governance procedures, extending beyond South Korea.

Limitations and Future Research

The governance guidelines bear inherent limitations. Foremost, they do not decisively address the liability associated with possible harm resulting from healthcare AI applications. In the case of healthcare AI research and application involving multiple parties, it is necessary to examine whether the harm caused can be assumed the same as the existing medical liability process. For example, if a patient is physically harmed in the process of utilizing a healthcare AI device, but it turns out to be a problem with the algorithm rather than the fault of the medical practitioner or device user, who should be held liable?

Navigating liability questions amidst the overlapping influences of various actors poses challenges (Kim 2017 ). While the governance of healthcare AI research needs to address the issue of liability, it is limited by the fact that the guidelines in the study focus on proposing an ethical model grounded in self-regulation, addressing the intricacies of liability remains a significant challenge. Moreover, the procedures are set to be adjusted according to each country’s regulatory procedures, which is because the procedures correspond to existing clinical research guidelines, but it is necessary to examine whether they can be properly operated in real-life situations. This is an area that requires empirical verification by applying the guidelines to actual healthcare AI research governance. Therefore, this paper calls for further research on the healthcare AI governance guidelines presented here to address the issues identified above, especially linking it the legal standard to regulation.

To address the identified limitations, researchers are actively engaged in ongoing efforts in education of AI researchers and the public, social communication, and the revision of the guidelines. These initiatives will ensure a comprehensive societal understanding and adoption of healthcare AI research ethics, encourage researchers and developers to accept the need to conduct research ethically, and thereby facilitate the operationalization of ethical governance systems at both institutional and national levels within the South Korean context. As a result, these endeavours will significantly contribute to the establishment of a robust ethical normative framework for healthcare AI research in this country.

Since the governance settings presented in this study are from the perspective of a specific country, it is necessary to collect the opinions of researchers and bioethicists from other countries through international discussions and reviews. In order to facilitate such discussions, this study aims to inform other countries about the governance system established in South Korea and, using this study as a starting point, collect multi-perspective and multi-disciplinary views on healthcare AI research governance that have not yet been organized and provide basic data on the establishment of cross-border healthcare AI research governance.

The aims of this study are to present a healthcare AI research governance system founded on the South Korean ‘Research Ethics Guidelines for AI Researchers in Healthcare’ and to elucidate each procedural step. The six-stage healthcare AI research governance framework mirrors the healthcare AI research and development process, and is designed in harmony with the existing clinical research management systems. This parallel structure facilitates the utilization of established research management resources and foster mutual understanding among researchers and institutions for conducting ethical research procedures. Nonetheless, the guidelines are likely to reflect the specificities of the Korean healthcare environment, emphasizing the need for further international dialogue and refinement.

Data Availability

The framework employed in our research is included in the English version of “Research Ethics Guidelines for Healthcare AI Researchers” (KNIH 2023 ). This document is currently in the process of being published. Upon its publication, we will promptly provide the relevant link.

Cross-Industry Standard Process for Data Mining

Knowledge Discovery in Database

Sample, Explore, Modify, Model, and Assess

Cognitive Project Management for AI

Cha, Hyun-Jae., and Junhewk Kim. 2022. The ethical approach to health data donation and sharing: From the process of human tissue donation. Bio, Ethics and Policy 6 (2): 101–137.

Google Scholar  

Gerke, Sara, Boris Babic, I. Theodoros Evgenious, and Glenn Cohen. 2020. The need for a system view to regulated artificial intelligence/machine learning-based software as medical device. NPJ Digital Medicine  3: 53. https://doi.org/10.1038/s41746-020-0262-2 .

Higgins, David, and Vince I. Madai. 2020. From bit to bedside: A practical framework for artificial intelligence product development in healthcare. Advanced Intelligent Systems 2 (10): 2000052. https://doi.org/10.1002/aisy.202000052 .

Article   Google Scholar  

Kim, Junhewk. 2017. Autonomous decision medical system and moral responsibility. Philosophy of Medicine 24: 147–182.

Kim, Hannah, Jung Im Lee, Jinah Sim, Yuri Lee, So Yoon Kim, Eun-Ae Kim, Soo Min Kim, and Junhewk Kim. 2023. Ethical guidelines for artificial intelligence research in healthcare: Introducing South Korean Perspectives. Korean Journal of Medicine and Law 31 (1): 85–110. https://doi.org/10.17215/kaml.2023.06.31.1.85 .

Korean National Institute of Health. 2023. Research ethics guidelines for AI researchers in healthcare . Cheongju: Korea Disease Control and Prevention Agency.

Leonelli, Sabina. 2016. Data-centric biology: A philosophical study . Chicago: The University of Chicago Press.

Book   Google Scholar  

Li, Hong, Gustavo Glusman, Hao Hu, Shankaracharya, Juan Caballero, and Robert Hubley, et al. 2014. Relationship estimation from whole-genome sequence data. PLoS Genetics 10(1): e1004144. http://dx.doi.org/10.1371/journal.pgen.1004144.

Metcalf, Jacob, and Kate Crawford. 2016. Where are human subjects in Big Data research? The emerging ethics divide. Big Data & Society 3 (1): 1–14. https://doi.org/10.1177/2053951716650211

Ministry of Food and Drug Safety of the Republic of Korea. 2023. 2022 Medical device license report . Cheongju: Ministry of Food and Drug Safety of the Republic of Korea.

Ministry of Health and Welfare of South Korea. 2022. Guidelines for utilization of healthcare data . Sejong: Ministry of Health and Welfare of South Korea.

National Commission for the Protection of Human Subjects of Biomedical & Behavioral Research. 1978. The Belmont report: Ethical principles and guidelines for the protection of human subjects of research . Bethesda, MD: National Commission for the Protection of Human Subjects of Biomedical & Behavioral Research.

Pianykh, Oleg S., Georg Langs, Marc Dewey, Dieter R. Enzmann, Christian J. Herold, and Stefan O. Schoenberg, et al. 2020. Continuous learning AI in radiology: Implementation principles and early application. Radiology 297 (1): 6–14. https://doi.org/10.1148/radiol.2020200038 .

Rajpurkjar, Pranav, Emma Chen, Oishi Banerjee, and Eric J. Topol. 2022. AI in health and medicine. Nature Medicine 28: 31–38. https://doi.org/10.1038/s41591-021-01614-0 .

Sujan, Mark, Cassius Smith-Frazer, Christina Malamateniou, Joseph Connor, Allison Garner, Harriet Unsworth, et al. 2023. Validation framework for the use of AI in healthcare: Overview of the new British standard BS30440. BMJ Health & Care Informatics 30: e100749. https://doi.org/10.1136/bmjhci-2023-100749 .

Throne, Robin. 2022. Adverse trends in data ethics: The AI Bill of Rights and Human Subjects Protections. SSRN , 30 November 2022. https://doi.org/10.2139/ssrn.4279922 .

Wu, Eric, Kevin Wu, Roxana Daneshjou, David Ouyang, Daniel E. Ho, and James Zou. 2021. How medical AI devices are evaluated: Limitations and recommendations from an analysis of FDA approvals. Nature Medicine 27: 582–584. https://doi.org/10.1038/s41591-021-01312-x .

Download references

Acknowledgements

The authors wish to thank Dr Jung-Im Lee and Dr Sumin Kim for their contribution in developing the guidelines. The first project (2022-ER0807-00) conducted consultation meetings of two panels of interdisciplinary expert participants from law, public health policy, ethics, AI, and patients group for four times from August, 2022, to February, 2023, and a public hearing at February 2023. We deeply express our gratitude for all participants for their valuable opinions.

This work was supported by the ‘Development of Ethics Guidelines and Education Program for the Use of Artificial Intelligent in Healthcare Research’ and ‘Operation of Education Program and Improvement of Ethics Guidelines for the Use of Artificial Intelligent in Healthcare Research’ from the Korean National Institutes of Health (Grant numbers: 2022-ER0807-00 and 2023-ER0808-00).

Author information

Authors and affiliations.

Department of Dental Education, College of Dentistry, Yonsei University, Seoul, South Korea

Junhewk Kim

Asian Institute for Bioethics and Health Law, Department of Medical Humanities and Social Sciences, College of Medicine, Yonsei University, Seoul, South Korea

So Yoon Kim & Hannah Kim

Center for Research Compliance, Ewha Womans University, Seoul, South Korea

Department of AI Convergence, Hallym University, Chuncheon, South Korea

Department of Health and Medical Information, Myongji College, Seoul, South Korea

You can also search for this author in PubMed   Google Scholar

Contributions

J. K. and H. K. were responsible for the conception, design, acquisition of data or analysis, and interpretation of data. J. K. was responsible for manuscript writing, subsequent revisions of the manuscript and funding (2023-ER0808-00). H. K. was responsible for reviewing the manuscript, funding (2022-ER0807-00), and developing the guidelines. S. Y. K., E. A. K., J. A. S., and Y. L. participated in developing the guidelines and reviewing the manuscript. All authors have read and agreed to the published version of the manuscript.

Corresponding author

Correspondence to Hannah Kim .

Ethics declarations

During the research, the main responsibilities of the funding agency included managing the project progress and making decisions regarding the publication of the guidelines and the agency had no role in the study design, data collection and analysis, preparation of the manuscript, and decision to publish it.

Ethics Approval

As this study did not involve human participants, ethics approval was not needed.

Consent to Participate

Not applicable.

Consent for Publication

Competing interests.

The authors declare no competing interests.

Additional information

Publisher's note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Kim, J., Kim, S.Y., Kim, EA. et al. Developing a Framework for Self-regulatory Governance in Healthcare AI Research: Insights from South Korea. ABR (2024). https://doi.org/10.1007/s41649-024-00281-w

Download citation

Received : 30 September 2023

Revised : 11 January 2024

Accepted : 12 January 2024

Published : 25 March 2024

DOI : https://doi.org/10.1007/s41649-024-00281-w

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Healthcare AI research
• Research ethics
• Research governance
• AI governance
• Self-regulation
• Interdisciplinary approach
• Find a journal
• Publish with us
• Track your research