critical appraisal tools literature reviews

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

• Advanced Search
• Journal List
• Can Commun Dis Rep
• v.43(9); 2017 Sep 7

Scientific writing

Critical appraisal toolkit (cat) for assessing multiple types of evidence.

1 Memorial University School of Nursing, St. John’s, NL

2 Centre for Communicable Diseases and Infection Control, Public Health Agency of Canada, Ottawa, ON

Contributor: Jennifer Kruse, Public Health Agency of Canada – Conceptualization and project administration

Healthcare professionals are often expected to critically appraise research evidence in order to make recommendations for practice and policy development. Here we describe the Critical Appraisal Toolkit (CAT) currently used by the Public Health Agency of Canada. The CAT consists of: algorithms to identify the type of study design, three separate tools (for appraisal of analytic studies, descriptive studies and literature reviews), additional tools to support the appraisal process, and guidance for summarizing evidence and drawing conclusions about a body of evidence. Although the toolkit was created to assist in the development of national guidelines related to infection prevention and control, clinicians, policy makers and students can use it to guide appraisal of any health-related quantitative research. Participants in a pilot test completed a total of 101 critical appraisals and found that the CAT was user-friendly and helpful in the process of critical appraisal. Feedback from participants of the pilot test of the CAT informed further revisions prior to its release. The CAT adds to the arsenal of available tools and can be especially useful when the best available evidence comes from non-clinical trials and/or studies with weak designs, where other tools may not be easily applied.

Introduction

Healthcare professionals, researchers and policy makers are often involved in the development of public health policies or guidelines. The most valuable guidelines provide a basis for evidence-based practice with recommendations informed by current, high quality, peer-reviewed scientific evidence. To develop such guidelines, the available evidence needs to be critically appraised so that recommendations are based on the "best" evidence. The ability to critically appraise research is, therefore, an essential skill for health professionals serving on policy or guideline development working groups.

Our experience with working groups developing infection prevention and control guidelines was that the review of relevant evidence went smoothly while the critical appraisal of the evidence posed multiple challenges. Three main issues were identified. First, although working group members had strong expertise in infection prevention and control or other areas relevant to the guideline topic, they had varying levels of expertise in research methods and critical appraisal. Second, the critical appraisal tools in use at that time focused largely on analytic studies (such as clinical trials), and lacked definitions of key terms and explanations of the criteria used in the studies. As a result, the use of these tools by working group members did not result in a consistent way of appraising analytic studies nor did the tools provide a means of assessing descriptive studies and literature reviews. Third, working group members wanted guidance on how to progress from assessing individual studies to summarizing and assessing a body of evidence.

To address these issues, a review of existing critical appraisal tools was conducted. We found that the majority of existing tools were design-specific, with considerable variability in intent, criteria appraised and construction of the tools. A systematic review reported that fewer than half of existing tools had guidelines for use of the tool and interpretation of the items ( 1 ). The well-known Grading of Recommendations Assessment, Development and Evaluation (GRADE) rating-of-evidence system and the Cochrane tools for assessing risk of bias were considered for use ( 2 ), ( 3 ). At that time, the guidelines for using these tools were limited, and the tools were focused primarily on randomized controlled trials (RCTs) and non-randomized controlled trials. For feasibility and ethical reasons, clinical trials are rarely available for many common infection prevention and control issues ( 4 ), ( 5 ). For example, there are no intervention studies assessing which practice restrictions, if any, should be placed on healthcare workers who are infected with a blood-borne pathogen. Working group members were concerned that if they used GRADE, all evidence would be rated as very low or as low quality or certainty, and recommendations based on this evidence may be interpreted as unconvincing, even if they were based on the best or only available evidence.

The team decided to develop its own critical appraisal toolkit. So a small working group was convened, led by an epidemiologist with expertise in research, methodology and critical appraisal, with the goal of developing tools to critically appraise studies informing infection prevention and control recommendations. This article provides an overview of the Critical Appraisal Toolkit (CAT). The full document, entitled Infection Prevention and Control Guidelines Critical Appraisal Tool Kit is available online ( 6 ).

Following a review of existing critical appraisal tools, studies informing infection prevention and control guidelines that were in development were reviewed to identify the types of studies that would need to be appraised using the CAT. A preliminary draft of the CAT was used by various guideline development working groups and iterative revisions were made over a two year period. A pilot test of the CAT was then conducted which led to the final version ( 6 ).

The toolkit is set up to guide reviewers through three major phases in the critical appraisal of a body of evidence: appraisal of individual studies; summarizing the results of the appraisals; and appraisal of the body of evidence.

Tools for critically appraising individual studies

The first step in the critical appraisal of an individual study is to identify the study design; this can be surprisingly problematic, since many published research studies are complex. An algorithm was developed to help identify whether a study was an analytic study, a descriptive study or a literature review (see text box for definitions). It is critical to establish the design of the study first, as the criteria for assessment differs depending on the type of study.

Definitions of the types of studies that can be analyzed with the Critical Appraisal Toolkit*

Analytic study: A study designed to identify or measure effects of specific exposures, interventions or risk factors. This design employs the use of an appropriate comparison group to test epidemiologic hypotheses, thus attempting to identify associations or causal relationships.

Descriptive study: A study that describes characteristics of a condition in relation to particular factors or exposure of interest. This design often provides the first important clues about possible determinants of disease and is useful for the formulation of hypotheses that can be subsequently tested using an analytic design.

Literature review: A study that analyzes critical points of a published body of knowledge. This is done through summary, classification and comparison of prior studies. With the exception of meta-analyses, which statistically re-analyze pooled data from several studies, these studies are secondary sources and do not report any new or experimental work.

* Public Health Agency of Canada. Infection Prevention and Control Guidelines Critical Appraisal Tool Kit ( 6 )

Separate algorithms were developed for analytic studies, descriptive studies and literature reviews to help reviewers identify specific designs within those categories. The algorithm below, for example, helps reviewers determine which study design was used within the analytic study category ( Figure 1 ). It is based on key decision points such as number of groups or allocation to group. The legends for the algorithms and supportive tools such as the glossary provide additional detail to further differentiate study designs, such as whether a cohort study was retrospective or prospective.

Abbreviations: CBA, controlled before-after; ITS, interrupted time series; NRCT, non-randomized controlled trial; RCT, randomized controlled trial; UCBA, uncontrolled before-after

Separate critical appraisal tools were developed for analytic studies, for descriptive studies and for literature reviews, with relevant criteria in each tool. For example, a summary of the items covered in the analytic study critical appraisal tool is shown in Table 1 . This tool is used to appraise trials, observational studies and laboratory-based experiments. A supportive tool for assessing statistical analysis was also provided that describes common statistical tests used in epidemiologic studies.

The descriptive study critical appraisal tool assesses different aspects of sampling, data collection, statistical analysis, and ethical conduct. It is used to appraise cross-sectional studies, outbreak investigations, case series and case reports.

The literature review critical appraisal tool assesses the methodology, results and applicability of narrative reviews, systematic reviews and meta-analyses.

After appraisal of individual items in each type of study, each critical appraisal tool also contains instructions for drawing a conclusion about the overall quality of the evidence from a study, based on the per-item appraisal. Quality is rated as high, medium or low. While a RCT is a strong study design and a survey is a weak design, it is possible to have a poor quality RCT or a high quality survey. As a result, the quality of evidence from a study is distinguished from the strength of a study design when assessing the quality of the overall body of evidence. A definition of some terms used to evaluate evidence in the CAT is shown in Table 2 .

* Considered strong design if there are at least two control groups and two intervention groups. Considered moderate design if there is only one control and one intervention group

Tools for summarizing the evidence

The second phase in the critical appraisal process involves summarizing the results of the critical appraisal of individual studies. Reviewers are instructed to complete a template evidence summary table, with key details about each study and its ratings. Studies are listed in descending order of strength in the table. The table simplifies looking across all studies that make up the body of evidence informing a recommendation and allows for easy comparison of participants, sample size, methods, interventions, magnitude and consistency of results, outcome measures and individual study quality as determined by the critical appraisal. These evidence summary tables are reviewed by the working group to determine the rating for the quality of the overall body of evidence and to facilitate development of recommendations based on evidence.

Rating the quality of the overall body of evidence

The third phase in the critical appraisal process is rating the quality of the overall body of evidence. The overall rating depends on the five items summarized in Table 2 : strength of study designs, quality of studies, number of studies, consistency of results and directness of the evidence. The various combinations of these factors lead to an overall rating of the strength of the body of evidence as strong, moderate or weak as summarized in Table 3 .

A unique aspect of this toolkit is that recommendations are not graded but are formulated based on the graded body of evidence. Actions are either recommended or not recommended; it is the strength of the available evidence that varies, not the strength of the recommendation. The toolkit does highlight, however, the need to re-evaluate new evidence as it becomes available especially when recommendations are based on weak evidence.

Pilot test of the CAT

Of 34 individuals who indicated an interest in completing the pilot test, 17 completed it. Multiple peer-reviewed studies were selected representing analytic studies, descriptive studies and literature reviews. The same studies were assigned to participants with similar content expertise. Each participant was asked to appraise three analytic studies, two descriptive studies and one literature review, using the appropriate critical appraisal tool as identified by the participant. For each study appraised, one critical appraisal tool and the associated tool-specific feedback form were completed. Each participant also completed a single general feedback form. A total of 101 of 102 critical appraisals were conducted and returned, with 81 tool-specific feedback forms and 14 general feedback forms returned.

The majority of participants (>85%) found the flow of each tool was logical and the length acceptable but noted they still had difficulty identifying the study designs ( Table 4 ).

* Number of tool-specific forms returned for total number of critical appraisals conducted

The vast majority of the feedback forms (86–93%) indicated that the different tools facilitated the critical appraisal process. In the assessment of consistency, however, only four of ten analytic studies appraised (40%), had complete agreement on the rating of overall study quality by participants, the other six studies had differences noted as mismatches. Four of the six studies with mismatches were observational studies. The differences were minor. None of the mismatches included a study that was rated as both high and low quality by different participants. Based on the comments provided by participants, most mismatches could likely have been resolved through discussion with peers. Mismatched ratings were not an issue for the descriptive studies and literature reviews. In summary, the pilot test provided useful feedback on different aspects of the toolkit. Revisions were made to address the issues identified from the pilot test and thus strengthen the CAT.

The Infection Prevention and Control Guidelines Critical Appraisal Tool Kit was developed in response to the needs of infection control professionals reviewing literature that generally did not include clinical trial evidence. The toolkit was designed to meet the identified needs for training in critical appraisal with extensive instructions and dictionaries, and tools applicable to all three types of studies (analytic studies, descriptive studies and literature reviews). The toolkit provided a method to progress from assessing individual studies to summarizing and assessing the strength of a body of evidence and assigning a grade. Recommendations are then developed based on the graded body of evidence. This grading system has been used by the Public Health Agency of Canada in the development of recent infection prevention and control guidelines ( 5 ), ( 7 ). The toolkit has also been used for conducting critical appraisal for other purposes, such as addressing a practice problem and serving as an educational tool ( 8 ), ( 9 ).

The CAT has a number of strengths. It is applicable to a wide variety of study designs. The criteria that are assessed allow for a comprehensive appraisal of individual studies and facilitates critical appraisal of a body of evidence. The dictionaries provide reviewers with a common language and criteria for discussion and decision making.

The CAT also has a number of limitations. The tools do not address all study designs (e.g., modelling studies) and the toolkit provides limited information on types of bias. Like the majority of critical appraisal tools ( 10 ), ( 11 ), these tools have not been tested for validity and reliability. Nonetheless, the criteria assessed are those indicated as important in textbooks and in the literature ( 12 ), ( 13 ). The grading scale used in this toolkit does not allow for comparison of evidence grading across organizations or internationally, but most reviewers do not need such comparability. It is more important that strong evidence be rated higher than weak evidence, and that reviewers provide rationales for their conclusions; the toolkit enables them to do so.

Overall, the pilot test reinforced that the CAT can help with critical appraisal training and can increase comfort levels for those with limited experience. Further evaluation of the toolkit could assess the effectiveness of revisions made and test its validity and reliability.

A frequent question regarding this toolkit is how it differs from GRADE as both distinguish stronger evidence from weaker evidence and use similar concepts and terminology. The main differences between GRADE and the CAT are presented in Table 5 . Key differences include the focus of the CAT on rating the quality of individual studies, and the detailed instructions and supporting tools that assist those with limited experience in critical appraisal. When clinical trials and well controlled intervention studies are or become available, GRADE and related tools from Cochrane would be more appropriate ( 2 ), ( 3 ). When descriptive studies are all that is available, the CAT is very useful.

Abbreviation: GRADE, Grading of Recommendations Assessment, Development and Evaluation

The Infection Prevention and Control Guidelines Critical Appraisal Tool Kit was developed in response to needs for training in critical appraisal, assessing evidence from a wide variety of research designs, and a method for going from assessing individual studies to characterizing the strength of a body of evidence. Clinician researchers, policy makers and students can use these tools for critical appraisal of studies whether they are trying to develop policies, find a potential solution to a practice problem or critique an article for a journal club. The toolkit adds to the arsenal of critical appraisal tools currently available and is especially useful in assessing evidence from a wide variety of research designs.

Authors’ Statement

DM – Conceptualization, methodology, investigation, data collection and curation and writing – original draft, review and editing

TO – Conceptualization, methodology, investigation, data collection and curation and writing – original draft, review and editing

KD – Conceptualization, review and editing, supervision and project administration

Acknowledgements

We thank the Infection Prevention and Control Expert Working Group of the Public Health Agency of Canada for feedback on the development of the toolkit, Lisa Marie Wasmund for data entry of the pilot test results, Katherine Defalco for review of data and cross-editing of content and technical terminology for the French version of the toolkit, Laurie O’Neil for review and feedback on early versions of the toolkit, Frédéric Bergeron for technical support with the algorithms in the toolkit and the Centre for Communicable Diseases and Infection Control of the Public Health Agency of Canada for review, feedback and ongoing use of the toolkit. We thank Dr. Patricia Huston, Canada Communicable Disease Report Editor-in-Chief, for a thorough review and constructive feedback on the draft manuscript.

Conflict of interest: None.

Funding: This work was supported by the Public Health Agency of Canada.

Conducting a Literature Review

• Literature Review
• Developing a Topic
• Planning Your Literature Review
• Developing a Search Strategy
• Managing Citations
• Critical Appraisal Tools
• Writing a Literature Review

Appraise Your Research Articles

The structure of a literature review should include the following :

• An overview of the subject, issue, or theory under consideration, along with the objectives of the literature review,
• Division of works under review into themes or categories [e.g. works that support a particular position, those against, and those offering alternative approaches entirely],
• An explanation of how each work is similar to and how it varies from the others,
• Conclusions as to which pieces are best considered in their argument, are most convincing of their opinions, and make the greatest contribution to the understanding and development of their area of research.

The critical evaluation of each work should consider :

• Provenance  -- what are the author's credentials? Are the author's arguments supported by evidence [e.g. primary historical material, case studies, narratives, statistics, recent scientific findings]?
• Methodology  -- were the techniques used to identify, gather, and analyze the data appropriate to addressing the research problem? Was the sample size appropriate? Were the results effectively interpreted and reported?
• Objectivity  -- is the author's perspective even-handed or prejudicial? Is contrary data considered or is certain pertinent information ignored to prove the author's point?
• Persuasiveness  -- which of the author's theses are most convincing or least convincing?
• Value  -- are the author's arguments and conclusions convincing? Does the work ultimately contribute in any significant way to an understanding of the subject?

Reviewing the Literature

While conducting a review of the literature, maximize the time you devote to writing this part of your paper by thinking broadly about what you should be looking for and evaluating. Review not just what the articles are saying, but how are they saying it.

Some questions to ask:

• How are they organizing their ideas?
• What methods have they used to study the problem?
• What theories have been used to explain, predict, or understand their research problem?
• What sources have they cited to support their conclusions?
• How have they used non-textual elements [e.g., charts, graphs, figures, etc.] to illustrate key points?
• When you begin to write your literature review section, you'll be glad you dug deeper into how the research was designed and constructed because it establishes a means for developing more substantial analysis and interpretation of the research problem.

Tools for Critical Appraisal

Now, that you have found articles based on your research question you can appraise the quality of those articles. These are resources you can use to appraise different study designs.

Centre for Evidence Based Medicine (Oxford)

University of Glasgow

"AFP uses the Strength-of-Recommendation Taxonomy (SORT), to label key recommendations in clinical review articles."

• SORT: Rating the Strength of Evidence    American Family Physician and other family medicine journals use the Strength of Recommendation Taxonomy (SORT) system for rating bodies of evidence for key clinical recommendations.

• The Interprofessional Health Sciences Library
• 123 Metro Boulevard
• Nutley, NJ 07110
• [email protected]
• Visiting Campus
• News and Events
• Parents and Families
• Web Accessibility
• Career Center
• Public Safety
• Accountability
• Privacy Statements
• Report a Problem
• Login to LibApps

• University of Texas Libraries
• UT Libraries

Systematic Reviews & Evidence Synthesis Methods

Critical appraisal.

• Types of Reviews
• Formulate Question
• Find Existing Reviews & Protocols
• Register a Protocol
• Searching Systematically
• Supplementary Searching
• Managing Results
• Deduplication
• Glossary of terms
• Librarian Support
• Video tutorials This link opens in a new window
• Systematic Review & Evidence Synthesis Boot Camp

Some reviews require a critical appraisal for each study that makes it through the screening process. This involves a risk of bias assessment and/or a quality assessment. The goal of these reviews is not just to find all of the studies, but to determine their methodological rigor, and therefore, their credibility.

"Critical appraisal is the balanced assessment of a piece of research, looking for its strengths and weaknesses and them coming to a balanced judgement about its trustworthiness and its suitability for use in a particular context." 1

It's important to consider the impact that poorly designed studies could have on your findings and to rule out inaccurate or biased work.

Selection of a valid critical appraisal tool, testing the tool with several of the selected studies, and involving two or more reviewers in the appraisal are good practices to follow.

1. Purssell E, McCrae N. How to Perform a Systematic Literature Review: A Guide for Healthcare Researchers, Practitioners and Students. 1st ed. Springer ;  2020.

Evaluation Tools

• The Appraisal of Guidelines for Research & Evaluation Instrument (AGREE II) The Appraisal of Guidelines for Research & Evaluation Instrument (AGREE II) was developed to address the issue of variability in the quality of practice guidelines.
• Centre for Evidence-Based Medicine (CEBM). Critical Appraisal Tools "contains useful tools and downloads for the critical appraisal of different types of medical evidence. Example appraisal sheets are provided together with several helpful examples."
• Critical Appraisal Skills Programme (CASP) Checklists Critical Appraisal checklists for many different study types
• Critical Review Form for Qualitative Studies Version 2, developed out of McMaster University
• Development of a critical appraisal tool to assess the quality of cross-sectional studies (AXIS) Downes MJ, Brennan ML, Williams HC, et al. Development of a critical appraisal tool to assess the quality of cross-sectional studies (AXIS). BMJ Open 2016;6:e011458. doi:10.1136/bmjopen-2016-011458
• Downs & Black Checklist for Assessing Studies Downs, S. H., & Black, N. (1998). The Feasibility of Creating a Checklist for the Assessment of the Methodological Quality Both of Randomised and Non-Randomised Studies of Health Care Interventions. Journal of Epidemiology and Community Health (1979-), 52(6), 377–384.
• GRADE The Grading of Recommendations Assessment, Development and Evaluation (GRADE) working group "has developed a common, sensible and transparent approach to grading quality (or certainty) of evidence and strength of recommendations."
• Grade Handbook Full handbook on the GRADE method for grading quality of evidence.
• MAGIC (Making GRADE the Irresistible choice) Clear succinct guidance in how to use GRADE
• Joanna Briggs Institute. Critical Appraisal Tools "JBI’s critical appraisal tools assist in assessing the trustworthiness, relevance and results of published papers." Includes checklists for 13 types of articles.
• Latitudes Network This is a searchable library of validity assessment tools for use in evidence syntheses. This website also provides access to training on the process of validity assessment.
• Mixed Methods Appraisal Tool A tool that can be used to appraise a mix of studies that are included in a systematic review - qualitative research, RCTs, non-randomized studies, quantitative studies, mixed methods studies.
• RoB 2 Tool Higgins JPT, Sterne JAC, Savović J, Page MJ, Hróbjartsson A, Boutron I, Reeves B, Eldridge S. A revised tool for assessing risk of bias in randomized trials In: Chandler J, McKenzie J, Boutron I, Welch V (editors). Cochrane Methods. Cochrane Database of Systematic Reviews 2016, Issue 10 (Suppl 1). dx.doi.org/10.1002/14651858.CD201601.
• ROBINS-I Risk of Bias for non-randomized (observational) studies or cohorts of interventions Sterne J A, Hernán M A, Reeves B C, Savović J, Berkman N D, Viswanathan M et al. ROBINS-I: a tool for assessing risk of bias in non-randomised studies of interventions BMJ 2016; 355 :i4919 doi:10.1136/bmj.i4919
• Scottish Intercollegiate Guidelines Network. Critical Appraisal Notes and Checklists "Methodological assessment of studies selected as potential sources of evidence is based on a number of criteria that focus on those aspects of the study design that research has shown to have a significant effect on the risk of bias in the results reported and conclusions drawn. These criteria differ between study types, and a range of checklists is used to bring a degree of consistency to the assessment process."
• The TREND Statement (CDC) Des Jarlais DC, Lyles C, Crepaz N, and the TREND Group. Improving the reporting quality of nonrandomized evaluations of behavioral and public health interventions: The TREND statement. Am J Public Health. 2004;94:361-366.
• Assembling the Pieces of a Systematic Reviews, Chapter 8: Evaluating: Study Selection and Critical Appraisal.
• How to Perform a Systematic Literature Review, Chapter: Critical Appraisal: Assessing the Quality of Studies.

Other library guides

• Duke University Medical Center Library. Systematic Reviews: Assess for Quality and Bias
• UNC Health Sciences Library. Systematic Reviews: Assess Quality of Included Studies
• Last Updated: May 16, 2024 11:05 AM
• URL: https://guides.lib.utexas.edu/systematicreviews

• Research article
• Open access
• Published: 16 September 2004

A systematic review of the content of critical appraisal tools

• Persis Katrak 1 ,
• Andrea E Bialocerkowski 2 ,
• Nicola Massy-Westropp 1 ,
• VS Saravana Kumar 1 &
• Karen A Grimmer 1  

BMC Medical Research Methodology volume  4 , Article number:  22 ( 2004 ) Cite this article

157k Accesses

208 Citations

14 Altmetric

Metrics details

Consumers of research (researchers, administrators, educators and clinicians) frequently use standard critical appraisal tools to evaluate the quality of published research reports. However, there is no consensus regarding the most appropriate critical appraisal tool for allied health research. We summarized the content, intent, construction and psychometric properties of published, currently available critical appraisal tools to identify common elements and their relevance to allied health research.

A systematic review was undertaken of 121 published critical appraisal tools sourced from 108 papers located on electronic databases and the Internet. The tools were classified according to the study design for which they were intended. Their items were then classified into one of 12 criteria based on their intent. Commonly occurring items were identified. The empirical basis for construction of the tool, the method by which overall quality of the study was established, the psychometric properties of the critical appraisal tools and whether guidelines were provided for their use were also recorded.

Eighty-seven percent of critical appraisal tools were specific to a research design, with most tools having been developed for experimental studies. There was considerable variability in items contained in the critical appraisal tools. Twelve percent of available tools were developed using specified empirical research. Forty-nine percent of the critical appraisal tools summarized the quality appraisal into a numeric summary score. Few critical appraisal tools had documented evidence of validity of their items, or reliability of use. Guidelines regarding administration of the tools were provided in 43% of cases.

Conclusions

There was considerable variability in intent, components, construction and psychometric properties of published critical appraisal tools for research reports. There is no "gold standard' critical appraisal tool for any study design, nor is there any widely accepted generic tool that can be applied equally well across study types. No tool was specific to allied health research requirements. Thus interpretation of critical appraisal of research reports currently needs to be considered in light of the properties and intent of the critical appraisal tool chosen for the task.

Peer Review reports

Consumers of research (clinicians, researchers, educators, administrators) frequently use standard critical appraisal tools to evaluate the quality and utility of published research reports [ 1 ]. Critical appraisal tools provide analytical evaluations of the quality of the study, in particular the methods applied to minimise biases in a research project [ 2 ]. As these factors potentially influence study results, and the way that the study findings are interpreted, this information is vital for consumers of research to ascertain whether the results of the study can be believed, and transferred appropriately into other environments, such as policy, further research studies, education or clinical practice. Hence, choosing an appropriate critical appraisal tool is an important component of evidence-based practice.

Although the importance of critical appraisal tools has been acknowledged [ 1 , 3 – 5 ] there appears to be no consensus regarding the 'gold standard' tool for any medical evidence. In addition, it seems that consumers of research are faced with a large number of critical appraisal tools from which to choose. This is evidenced by the recent report by the Agency for Health Research Quality in which 93 critical appraisal tools for quantitative studies were identified [ 6 ]. Such choice may pose problems for research consumers, as dissimilar findings may well be the result when different critical appraisal tools are used to evaluate the same research report [ 6 ].

Critical appraisal tools can be broadly classified into those that are research design-specific and those that are generic. Design-specific tools contain items that address methodological issues that are unique to the research design [ 5 , 7 ]. This precludes comparison however of the quality of different study designs [ 8 ]. To attempt to overcome this limitation, generic critical appraisal tools have been developed, in an attempt to enhance the ability of research consumers to synthesise evidence from a range of quantitative and or qualitative study designs (for instance [ 9 ]). There is no evidence that generic critical appraisal tools and design-specific tools provide a comparative evaluation of research designs.

Moreover, there appears to be little consensus regarding the most appropriate items that should be contained within any critical appraisal tool. This paper is concerned primarily with critical appraisal tools that address the unique properties of allied health care and research [ 10 ]. This approach was taken because of the unique nature of allied health contacts with patients, and because evidence-based practice is an emerging area in allied health [ 10 ]. The availability of so many critical appraisal tools (for instance [ 6 ]) may well prove daunting for allied health practitioners who are learning to critically appraise research in their area of interest. For the purposes of this evaluation, allied health is defined as encompassing "...all occasions of service to non admitted patients where services are provided at units/clinics providing treatment/counseling to patients. These include units primarily concerned with physiotherapy, speech therapy, family panning, dietary advice, optometry occupational therapy..." [ 11 ].

The unique nature of allied health practice needs to be considered in allied health research. Allied health research thus differs from most medical research, with respect to:

• the paradigm underpinning comprehensive and clinically-reasoned descriptions of diagnosis (including validity and reliability). An example of this is in research into low back pain, where instead of diagnosis being made on location and chronicity of pain (as is common) [ 12 ], it would be made on the spinal structure and the nature of the dysfunction underpinning the symptoms, which is arrived at by a staged and replicable clinical reasoning process [ 10 , 13 ].

• the frequent use of multiple interventions within the one contact with the patient (an occasion of service), each of which requires appropriate description in terms of relationship to the diagnosis, nature, intensity, frequency, type of instruction provided to the patient, and the order in which the interventions were applied [ 13 ]

• the timeframe and frequency of contact with the patient (as many allied health disciplines treat patients in episodes of care that contain multiple occasions of service, and which can span many weeks, or even years in the case of chronic problems [ 14 ])

• measures of outcome, including appropriate methods and timeframes of measuring change in impairment, function, disability and handicap that address the needs of different stakeholders (patients, therapists, funders etc) [ 10 , 12 , 13 ].

Search strategy

In supplementary data [see additional file 1 ].

Data organization and extraction

Two independent researchers (PK, NMW) participated in all aspects of this review, and they compared and discussed their findings with respect to inclusion of critical appraisal tools, their intent, components, data extraction and item classification, construction and psychometric properties. Disagreements were resolved by discussion with a third member of the team (KG).

Data extraction consisted of a four-staged process. First, identical replica critical appraisal tools were identified and removed prior to analysis. The remaining critical appraisal tools were then classified according to the study design for which they were intended to be used [ 1 , 2 ]. The scientific manner in which the tools had been constructed was classified as whether an empirical research approach has been used, and if so, which type of research had been undertaken. Finally, the items contained in each critical appraisal tool were extracted and classified into one of eleven groups, which were based on the criteria described by Clarke and Oxman [ 4 ] as:

• Study aims and justification

• Methodology used , which encompassed method of identification of relevant studies and adherence to study protocol;

• Sample selection , which ranged from inclusion and exclusion criteria, to homogeneity of groups;

• Method of randomization and allocation blinding;

• Attrition : response and drop out rates;

• Blinding of the clinician, assessor, patient and statistician as well as the method of blinding;

• Outcome measure characteristics;

• Intervention or exposure details;

• Method of data analyses ;

• Potential sources of bias ; and

• Issues of external validity , which ranged from application of evidence to other settings to the relationship between benefits, cost and harm.

An additional group, " miscellaneous ", was used to describe items that could not be classified into any of the groups listed above.

Data synthesis

Data was synthesized using MS Excel spread sheets as well as narrative format by describing the number of critical appraisal tools per study design and the type of items they contained. Descriptions were made of the method by which the overall quality of the study was determined, evidence regarding the psychometric properties of the tools (validity and reliability) and whether guidelines were provided for use of the critical appraisal tool.

One hundred and ninety-three research reports that potentially provided a description of a critical appraisal tool (or process) were identified from the search strategy. Fifty-six of these papers were unavailable for review due to outdated Internet links, or inability to source the relevant journal through Australian university and Government library databases. Of the 127 papers retrieved, 19 were excluded from this review, as they did not provide a description of the critical appraisal tool used, or were published in languages other than English. As a result, 108 papers were reviewed, which yielded 121 different critical appraisal tools [ 1 – 5 , 7 , 9 , 15 – 102 , 116 ].

Empirical basis for tool construction

We identified 14 instruments (12% all tools) which were reported as having been constructed using a specified empirical approach [ 20 , 29 , 30 , 32 , 35 , 40 , 49 , 51 , 70 – 72 , 79 , 103 , 116 ]. The empirical research reflected descriptive and/or qualitative approaches, these being critical review of existing tools [ 40 , 72 ], Delphi techniques to identify then refine data items [ 32 , 51 , 71 ], questionnaires and other forms of written surveys to identify and refine data items [ 70 , 79 , 103 ], facilitated structured consensus meetings [ 20 , 29 , 30 , 35 , 40 , 49 , 70 , 72 , 79 , 116 ], and pilot validation testing [ 20 , 40 , 72 , 103 , 116 ]. In all the studies which reported developing critical appraisal tools using a consensus approach, a range of stakeholder input was sought, reflecting researchers and clinicians in a range of health disciplines, students, educators and consumers. There were a further 31 papers which cited other studies as the source of the tool used in the review, but which provided no information on why individual items had been chosen, or whether (or how) they had been modified. Moreover, for 21 of these tools, the cited sources of the critical appraisal tool did not report the empirical basis on which the tool had been constructed.

Critical appraisal tools per study design

Seventy-eight percent (N = 94) of the critical appraisal tools were developed for use on primary research [ 1 – 5 , 7 , 9 , 18 , 19 , 25 – 27 , 34 , 37 – 41 ], while the remainder (N = 26) were for secondary research (systematic reviews and meta-analyses) [ 2 – 5 , 15 – 36 , 116 ]. Eighty-seven percent (N = 104) of all critical appraisal tools were design-specific [ 2 – 5 , 7 , 9 , 15 – 90 ], with over one third (N = 45) developed for experimental studies (randomized controlled trials, clinical trials) [ 2 – 4 , 25 – 27 , 34 , 37 – 73 ]. Sixteen critical appraisal tools were generic. Of these, six were developed for use on both experimental and observational studies [ 9 , 91 – 95 ], whereas 11 were purported to be useful for any qualitative and quantitative research design [ 1 , 18 , 41 , 96 – 102 , 116 ] (see Figure 1 , Table 1 ).

Number of critical appraisal tools per study design [1,2]

Critical appraisal items

One thousand, four hundred and seventy five items were extracted from these critical appraisal tools. After grouping like items together, 173 different item types were identified, with the most frequently reported items being focused towards assessing the external validity of the study (N = 35) and method of data analyses (N = 28) (Table 2 ). The most frequently reported items across all critical appraisal tools were:

Eligibility criteria (inclusion/exclusion criteria) (N = 63)

Appropriate statistical analyses (N = 47)

Random allocation of subjects (N = 43)

Consideration of outcome measures used (N = 43)

Sample size justification/power calculations (N = 39)

Study design reported (N = 36)

Assessor blinding (N = 36)

Design-specific critical appraisal tools

Systematic reviews.

Eighty-seven different items were extracted from the 26 critical appraisal tools, which were designed to evaluate the quality of systematic reviews. These critical appraisal tools frequently contained items regarding data analyses and issues of external validity (Tables 2 and 3 ).

Items assessing data analyses were focused to the methods used to summarize the results, assessment of sensitivity of results and whether heterogeneity was considered, whereas the nature of reporting of the main results, interpretation of them and their generalizability were frequently used to assess the external validity of the study findings. Moreover, systematic review critical appraisal tools tended to contain items such as identification of relevant studies, search strategy used, number of studies included and protocol adherence, that would not be relevant for other study designs. Blinding and randomisation procedures were rarely included in these critical appraisal tools.

Experimental studies

One hundred and twenty thirteen different items were extracted from the 45 experimental critical appraisal tools. These items most frequently assessed aspects of data analyses and blinding (Tables 1 and 2 ). Data analyses items were focused on whether appropriate statistical analysis was performed, whether a sample size justification or power calculation was provided and whether side effects of the intervention were recorded and analysed. Blinding was focused on whether the participant, clinician and assessor were blinded to the intervention.

Diagnostic studies

Forty-seven different items were extracted from the seven diagnostic critical appraisal tools. These items frequently addressed issues involving data analyses, external validity of results and sample selection that were specific to diagnostic studies (whether the diagnostic criteria were defined, definition of the "gold" standard, the calculation of sensitivity and specificity) (Tables 1 and 2 ).

Observational studies

Seventy-four different items were extracted from the 19 critical appraisal tools for observational studies. These items primarily focused on aspects of data analyses (see Tables 1 and 2 , such as whether confounders were considered in the analysis, whether a sample size justification or power calculation was provided and whether appropriate statistical analyses were preformed.

Qualitative studies

Thirty-six different items were extracted from the seven qualitative study critical appraisal tools. The majority of these items assessed issues regarding external validity, methods of data analyses and the aims and justification of the study (Tables 1 and 2 ). Specifically, items were focused to whether the study question was clearly stated, whether data analyses were clearly described and appropriate, and application of the study findings to the clinical setting. Qualitative critical appraisal tools did not contain items regarding sample selection, randomization, blinding, intervention or bias, perhaps because these issues are not relevant to the qualitative paradigm.

Generic critical appraisal tools

Experimental and observational studies.

Forty-two different items were extracted from the six critical appraisal tools that could be used to evaluate experimental and observational studies. These tools most frequently contained items that addressed aspects of sample selection (such as inclusion/exclusion criteria of participants, homogeneity of participants at baseline) and data analyses (such as whether appropriate statistical analyses were performed, whether a justification of the sample size or power calculation were provided).

All study designs

Seventy-eight different items were contained in the ten critical appraisal tools that could be used for all study designs (quantitative and qualitative). The majority of these items focused on whether appropriate data analyses were undertaken (such as whether confounders were considered in the analysis, whether a sample size justification or power calculation was provided and whether appropriate statistical analyses were preformed) and external validity issues (generalization of results to the population, value of the research findings) (see Tables 1 and 2 ).

Allied health critical appraisal tools

We found no critical appraisal instrument specific to allied health research, despite finding at least seven critical appraisal instruments associated with allied health topics (mostly physiotherapy management of orthopedic conditions) [ 37 , 39 , 52 , 58 , 59 , 65 ]. One critical appraisal development group proposed two instruments [ 9 ], specific to quantitative and qualitative research respectively. The core elements of allied health research quality (specific diagnosis criteria, intervention descriptions, nature of patient contact and appropriate outcome measures) were not addressed in any one tool sourced for this evaluation. We identified 152 different ways of considering quality reporting of outcome measures in the 121 critical appraisal tools, and 81 ways of considering description of interventions. Very few tools which were not specifically targeted to diagnostic studies (less than 10% of the remaining tools) addressed diagnostic criteria. The critical appraisal instrument that seemed most related to allied health research quality [ 39 ] sought comprehensive evaluation of elements of intervention and outcome, however this instrument was relevant only to physiotherapeutic orthopedic experimental research.

Overall study quality

Forty-nine percent (N = 58) of critical appraisal tools summarised the results of the quality appraisal into a single numeric summary score [ 5 , 7 , 15 – 25 , 37 – 59 , 74 – 77 , 80 – 83 , 87 , 91 – 93 , 96 , 97 ] (Figure 2 ). This was achieved by one of two methods:

Number of critical appraisal tools with, and without, summary quality scores

An equal weighting system, where one point was allocated to each item fulfilled; or

A weighted system, where fulfilled items were allocated various points depending on their perceived importance.

However, there was no justification provided for any of the scoring systems used. In the remaining critical appraisal tools (N = 62), a single numerical summary score was not provided [ 1 – 4 , 9 , 25 – 36 , 60 – 73 , 78 , 79 , 84 – 90 , 94 , 95 , 98 – 102 ]. This left the research consumer to summarize the results of the appraisal in a narrative manner, without the assistance of a standard approach.

Psychometric properties of critical appraisal tools

Few critical appraisal tools had documented evidence of their validity and reliability. Face validity was established in nine critical appraisal tools, seven of which were developed for use on experimental studies [ 38 , 40 , 45 , 49 , 51 , 63 , 70 ] and two for systematic reviews [ 32 , 103 ]. Intra-rater reliability was established for only one critical appraisal tool as part of its empirical development process [ 40 ], whereas inter-rater reliability was reported for two systematic review tools [ 20 , 36 ] (for one of these as part of the developmental process [ 20 ]) and seven experimental critical appraisal tools [ 38 , 40 , 45 , 51 , 55 , 56 , 63 ] (for two of these as part of the developmental process [ 40 , 51 ]).

Critical appraisal tool guidelines

Forty-three percent (N = 52) of critical appraisal tools had guidelines that informed the user of the interpretation of each item contained within them (Table 2 ). These guidelines were most frequently in the form of a handbook or published paper (N = 31) [ 2 , 4 , 9 , 15 , 20 , 25 , 28 , 29 , 31 , 36 , 37 , 41 , 50 , 64 – 67 , 69 , 80 , 84 – 87 , 89 , 90 , 95 , 100 , 116 ], whereas in 14 critical appraisal tools explanations accompanied each item [ 16 , 26 , 27 , 40 , 49 , 51 , 57 , 59 , 79 , 83 , 91 , 102 ].

Our search strategy identified a large number of published critical appraisal tools that are currently available to critically appraise research reports. There was a distinct lack of information on tool development processes in most cases. Many of the tools were reported to be modifications of other published tools, or reflected specialty concerns in specific clinical or research areas, without attempts to justify inclusion criteria. Less than 10 of these tools were relevant to evaluation of the quality of allied health research, and none of these were based on an empirical research approach. We are concerned that although our search was systematic and extensive [ 104 , 105 ], our broad key words and our lack of ready access to 29% of potentially useful papers (N = 56) potentially constrained us from identifying all published critical appraisal tools. However, consumers of research seeking critical appraisal instruments are not likely to seek instruments from outdated Internet links and unobtainable journals, thus we believe that we identified the most readily available instruments. Thus, despite the limitations on sourcing all possible tools, we believe that this paper presents a useful synthesis of the readily available critical appraisal tools.

The majority of the critical appraisal tools were developed for a specific research design (87%), with most designed for use on experimental studies (38% of all critical appraisal tools sourced). This finding is not surprising as, according to the medical model, experimental studies sit at or near the top of the hierarchy of evidence [ 2 , 8 ]. In recent years, allied health researchers have strived to apply the medical model of research to their own discipline by conducting experimental research, often by using the randomized controlled trial design [ 106 ]. This trend may be the reason for the development of experimental critical appraisal tools reported in allied health-specific research topics [ 37 , 39 , 52 , 58 , 59 , 65 ].

We also found a considerable number of critical appraisal tools for systematic reviews (N = 26), which reflects the trend to synthesize research evidence to make it relevant for clinicians [ 105 , 107 ]. Systematic review critical appraisal tools contained unique items (such as identification of relevant studies, search strategy used, number of studies included, protocol adherence) compared with tools used for primary studies, a reflection of the secondary nature of data synthesis and analysis.

In contrast, we identified very few qualitative study critical appraisal tools, despite the presence of many journal-specific guidelines that outline important methodological aspects required in a manuscript submitted for publication [ 108 – 110 ]. This finding may reflect the more traditional, quantitative focus of allied health research [ 111 ]. Alternatively, qualitative researchers may view the robustness of their research findings in different terms compared with quantitative researchers [ 112 , 113 ]. Hence the use of critical appraisal tools may be less appropriate for the qualitative paradigm. This requires further consideration.

Of the small number of generic critical appraisal tools, we found few that could be usefully applied (to any health research, and specifically to the allied health literature), because of the generalist nature of their items, variable interpretation (and applicability) of items across research designs, and/or lack of summary scores. Whilst these types of tools potentially facilitate the synthesis of evidence across allied health research designs for clinicians, their lack of specificity in asking the 'hard' questions about research quality related to research design also potentially precludes their adoption for allied health evidence-based practice. At present, the gold standard study design when synthesizing evidence is the randomized controlled trial [ 4 ], which underpins our finding that experimental critical appraisal tools predominated in the allied health literature [ 37 , 39 , 52 , 58 , 59 , 65 ]. However, as more systematic literature reviews are undertaken on allied health topics, it may become more accepted that evidence in the form of other research design types requires acknowledgement, evaluation and synthesis. This may result in the development of more appropriate and clinically useful allied health critical appraisal tools.

A major finding of our study was the volume and variation in available critical appraisal tools. We found no gold standard critical appraisal tool for any type of study design. Therefore, consumers of research are faced with frustrating decisions when attempting to select the most appropriate tool for their needs. Variable quality evaluations may be produced when different critical appraisal tools are used on the same literature [ 6 ]. Thus, interpretation of critical analysis must be carefully considered in light of the critical appraisal tool used.

The variability in the content of critical appraisal tools could be accounted for by the lack of any empirical basis of tool construction, established validity of item construction, and the lack of a gold standard against which to compare new critical tools. As such, consumers of research cannot be certain that the content of published critical appraisal tools reflect the most important aspects of the quality of studies that they assess [ 114 ]. Moreover, there was little evidence of intra- or inter-rater reliability of the critical appraisal tools. Coupled with the lack of protocols for use, this may mean that critical appraisers could interpret instrument items in different ways over repeated occasions of use. This may produce variable results [123].

Based on the findings of this evaluation, we recommend that consumers of research should carefully select critical appraisal tools for their needs. The selected tools should have published evidence of the empirical basis for their construction, validity of items and reliability of interpretation, as well as guidelines for use, so that the tools can be applied and interpreted in a standardized manner. Our findings highlight the need for consensus to be reached regarding the important and core items for critical appraisal tools that will produce a more standardized environment for critical appraisal of research evidence. As a consequence, allied health research will specifically benefit from having critical appraisal tools that reflect best practice research approaches which embed specific research requirements of allied health disciplines.

National Health and Medical Research Council: How to Review the Evidence: Systematic Identification and Review of the Scientific Literature. Canberra. 2000

Google Scholar  

National Health and Medical Research Council: How to Use the Evidence: Assessment and Application of Scientific Evidence. Canberra. 2000

Joanna Briggs Institute. [ http://www.joannabriggs.edu.au ]

Clarke M, Oxman AD: Cochrane Reviewer's Handbook 4.2.0. 2003, Oxford: The Cochrane Collaboration

Crombie IK: The Pocket Guide to Critical Appraisal: A Handbook for Health Care Professionals. 1996, London: BMJ Publishing Group

Agency for Healthcare Research and Quality: Systems to Rate the Strength of Scientific Evidence. Evidence Report/Technology Assessment No. 47, Publication No. 02-E016. Rockville. 2002

Elwood JM: Critical Appraisal of Epidemiological Studies and Clinical Trials. 1998, Oxford: Oxford University Press, 2

Sackett DL, Richardson WS, Rosenberg W, Haynes RB: Evidence Based Medicine. How to Practice and Teach EBM. 2000, London: Churchill Livingstone

Critical literature reviews. [ http://www.cotfcanada.org/cotf_critical.htm ]

Bialocerkowski AE, Grimmer KA, Milanese SF, Kumar S: Application of current research evidence to clinical physiotherapy practice. J Allied Health Res Dec.

The National Health Data Dictionary – Version 10. http://www.aihw.gov.au/publications/hwi/nhdd12/nhdd12-v1.pdf and http://www.aihw.gov.au/publications/hwi/nhdd12/nhdd12-v2.pdf

Grimmer K, Bowman P, Roper J: Episodes of allied health outpatient care: an investigation of service delivery in acute public hospital settings. Disability and Rehabilitation. 2000, 22 (1/2): 80-87.

CAS   PubMed   Google Scholar  

Grimmer K, Milanese S, Bialocerkowski A: Clinical guidelines for low back pain: A physiotherapy perspective. Physiotherapy Canada. 2003, 55 (4): 1-9.

Grimmer KA, Milanese S, Bialocerkowski AE, Kumar S: Producing and implementing evidence in clinical practice: the therapies' dilemma. Physiotherapy. 2004,

Greenhalgh T: How to read a paper: papers that summarize other papers (systematic reviews and meta-analysis). BMJ. 1997, 315: 672-675.

CAS   PubMed   PubMed Central   Google Scholar  

Auperin A, Pignon J, Poynard T: Review article: critical review of meta-analysis of randomised clinical trials in hepatogastroenterology. Alimentary Pharmacol Therapeutics. 1997, 11: 215-225. 10.1046/j.1365-2036.1997.131302000.x.

CAS   Google Scholar  

Barnes DE, Bero LA: Why review articles on the health effects of passive smoking reach different conclusions. J Am Med Assoc. 1998, 279: 1566-1570. 10.1001/jama.279.19.1566.

Beck CT: Use of meta-analysis as a teaching strategy in nursing research courses. J Nurs Educat. 1997, 36: 87-90.

Carruthers SG, Larochelle P, Haynes RB, Petrasovits A, Schiffrin EL: Report of the Canadian Hypertension Society Consensus Conference: 1. Introduction. Can Med Assoc J. 1993, 149: 289-293.

Oxman AD, Guyatt GH, Singer J, Goldsmith CH, Hutchinson BG, Milner RA, Streiner DL: Agreement among reviewers of review articles. J Clin Epidemiol. 1991, 44: 91-98. 10.1016/0895-4356(91)90205-N.

Sacks HS, Reitman D, Pagano D, Kupelnick B: Meta-analysis: an update. Mount Sinai Journal of Medicine. 1996, 63: 216-224.

Smith AF: An analysis of review articles published in four anaesthesia journals. Can J Anaesth. 1997, 44: 405-409.

L'Abbe KA, Detsky AS, O'Rourke K: Meta-analysis in clinical research. Ann Intern Med. 1987, 107: 224-233.

PubMed   Google Scholar  

Mulrow CD, Antonio S: The medical review article: state of the science. Ann Intern Med. 1987, 106: 485-488.

Continuing Professional Development: A Manual for SIGN Guideline Developers. [ http://www.sign.ac.uk ]

Learning and Development Public Health Resources Unit. [ http://www.phru.nhs.uk/ ]

FOCUS Critical Appraisal Tool. [ http://www.focusproject.org.uk ]

Cook DJ, Sackett DL, Spitzer WO: Methodologic guidelines for systematic reviews of randomized control trials in health care from the Potsdam Consultation on meta-analysis. J Clin Epidemiol. 1995, 48: 167-171. 10.1016/0895-4356(94)00172-M.

Cranney A, Tugwell P, Shea B, Wells G: Implications of OMERACT outcomes in arthritis and osteoporosis for Cochrane metaanalysis. J Rheumatol. 1997, 24: 1206-1207.

Guyatt GH, Sackett DL, Sinclair JC, Hoyward R, Cook DJ, Cook RJ: User's guide to the medical literature. IX. A method for grading health care recommendations. J Am Med Assoc. 1995, 274: 1800-1804. 10.1001/jama.274.22.1800.

Gyorkos TW, Tannenbaum TN, Abrahamowicz M, Oxman AD, Scott EAF, Milson ME, Rasooli Iris, Frank JW, Riben PD, Mathias RG: An approach to the development of practice guidelines for community health interventions. Can J Public Health. 1994, 85: S8-13.

Moher D, Cook DJ, Eastwood S, Olkin I, Rennie D, Stroup DF: Improving the quality of reports of meta-analyses of randomised controlled trials: the QUOROM statement. Quality of reporting of meta-analyses. Lancet. 1999, 354: 1896-1900. 10.1016/S0140-6736(99)04149-5.

Oxman AD, Cook DJ, Guyatt GH: Users' guides to the medical literature. VI. How to use an overview. Evidence-Based Medicine Working Group. J Am Med Assoc. 1994, 272: 1367-1371. 10.1001/jama.272.17.1367.

Pogue J, Yusuf S: Overcoming the limitations of current meta-analysis of randomised controlled trials. Lancet. 1998, 351: 47-52. 10.1016/S0140-6736(97)08461-4.

Stroup DF, Berlin JA, Morton SC, Olkin I, Williamson GD, Rennie D, Moher D, Becker BJ, Sipe TA, Thacker SB: Meta-analysis of observational studies in epidemiology: a proposal for reporting. Meta-analysis of observational studies in epidemiology (MOOSE) group. J Am Med Assoc. 2000, 283: 2008-2012. 10.1001/jama.283.15.2008.

Irwig L, Tosteson AN, Gatsonis C, Lau J, Colditz G, Chalmers TC, Mostellar F: Guidelines for meta-analyses evaluating diagnostic tests. Ann Intern Med. 1994, 120: 667-676.

Moseley AM, Herbert RD, Sherrington C, Maher CG: Evidence for physiotherapy practice: A survey of the Physiotherapy Evidence Database. Physiotherapy Evidence Database (PEDro). Australian Journal of Physiotherapy. 2002, 48: 43-50.

Cho MK, Bero LA: Instruments for assessing the quality of drug studies published in the medical literature. J Am Med Assoc. 1994, 272: 101-104. 10.1001/jama.272.2.101.

De Vet HCW, De Bie RA, Van der Heijden GJ, Verhagen AP, Sijpkes P, Kipschild PG: Systematic reviews on the basis of methodological criteria. Physiotherapy. 1997, 83: 284-289.

Downs SH, Black N: The feasibility of creating a checklist for the assessment of the methodological quality both of randomised and non-randomised studies of health care interventions. J Epidemiol Community Health. 1998, 52: 377-384.

Evans M, Pollock AV: A score system for evaluating random control clinical trials of prophylaxis of abdominal surgical wound infection. Br J Surg. 1985, 72: 256-260.

Fahey T, Hyde C, Milne R, Thorogood M: The type and quality of randomized controlled trials (RCTs) published in UK public health journals. J Public Health Med. 1995, 17: 469-474.

Gotzsche PC: Methodology and overt and hidden bias in reports of 196 double-blind trials of nonsteroidal antiinflammatory drugs in rheumatoid arthritis. Control Clin Trials. 1989, 10: 31-56. 10.1016/0197-2456(89)90017-2.

Imperiale TF, McCullough AJ: Do corticosteroids reduce mortality from alcoholic hepatitis? A meta-analysis of the randomized trials. Ann Int Med. 1990, 113: 299-307.

Jadad AR, Moore RA, Carroll D, Jenkinson C, Reynolds DJ, Gavaghan DJ, McQuay HJ: Assessing the quality of reports of randomized clinical trials: is blinding necessary?. Control Clin Trials. 1996, 17: 1-12. 10.1016/0197-2456(95)00134-4.

Khan KS, Daya S, Collins JA, Walter SD: Empirical evidence of bias in infertility research: overestimation of treatment effect in crossover trials using pregnancy as the outcome measure. Fertil Steril. 1996, 65: 939-945.

Kleijnen J, Knipschild P, ter Riet G: Clinical trials of homoeopathy. BMJ. 1991, 302: 316-323.

Liberati A, Himel HN, Chalmers TC: A quality assessment of randomized control trials of primary treatment of breast cancer. J Clin Oncol. 1986, 4: 942-951.

Moher D, Schulz KF, Altman DG, for the CONSORT Group: The CONSORT statement: revised recommendations for improving the quality of reports of parallel-group randomized trials. J Am Med Assoc. 2001, 285: 1987-1991. 10.1001/jama.285.15.1987.

Reisch JS, Tyson JE, Mize SG: Aid to the evaluation of therapeutic studies. Pediatrics. 1989, 84: 815-827.

Sindhu F, Carpenter L, Seers K: Development of a tool to rate the quality assessment of randomized controlled trials using a Delphi technique. J Advanced Nurs. 1997, 25: 1262-1268. 10.1046/j.1365-2648.1997.19970251262.x.

Van der Heijden GJ, Van der Windt DA, Kleijnen J, Koes BW, Bouter LM: Steroid injections for shoulder disorders: a systematic review of randomized clinical trials. Br J Gen Pract. 1996, 46: 309-316.

Van Tulder MW, Koes BW, Bouter LM: Conservative treatment of acute and chronic nonspecific low back pain. A systematic review of randomized controlled trials of the most common interventions. Spine. 1997, 22: 2128-2156. 10.1097/00007632-199709150-00012.

Garbutt JC, West SL, Carey TS, Lohr KN, Crews FT: Pharmacotherapy for Alcohol Dependence. Evidence Report/Technology Assessment No. 3, AHCPR Publication No. 99-E004. Rockville. 1999

Oremus M, Wolfson C, Perrault A, Demers L, Momoli F, Moride Y: Interarter reliability of the modified Jadad quality scale for systematic reviews of Alzheimer's disease drug trials. Dement Geriatr Cognit Disord. 2001, 12: 232-236. 10.1159/000051263.

Clark O, Castro AA, Filho JV, Djubelgovic B: Interrater agreement of Jadad's scale. Annual Cochrane Colloqium Abstracts. 2001, [ http://www.biomedcentral.com/abstracts/COCHRANE/1/op031 ]October Lyon

Jonas W, Anderson RL, Crawford CC, Lyons JS: A systematic review of the quality of homeopathic clinical trials. BMC Alternative Medicine. 2001, 1: 12-10.1186/1472-6882-1-12.

Van Tulder M, Malmivaara A, Esmail R, Koes B: Exercises therapy for low back pain: a systematic review within the framework of the Cochrane Collaboration back review group. Spine. 2000, 25: 2784-2796. 10.1097/00007632-200011010-00011.

Van Tulder MW, Ostelo R, Vlaeyen JWS, Linton SJ, Morley SJ, Assendelft WJJ: Behavioral treatment for chronic low back pain: a systematic review within the framework of the cochrane back. Spine. 2000, 25: 2688-2699. 10.1097/00007632-200010150-00024.

Aronson N, Seidenfeld J, Samson DJ, Aronson N, Albertson PC, Bayoumi AM, Bennett C, Brown A, Garber ABA, Gere M, Hasselblad V, Wilt T, Ziegler MPHK, Pharm D: Relative Effectiveness and Cost Effectiveness of Methods of Androgen Suppression in the Treatment of Advanced Prostate Cancer. Evidence Report/Technology Assessment No. 4, AHCPR Publication No.99-E0012. Rockville. 1999

Chalmers TC, Smith H, Blackburn B, Silverman B, Schroeder B, Reitman D, Ambroz A: A method for assessing the quality of a randomized control trial. Control Clin Trials. 1981, 2: 31-49. 10.1016/0197-2456(81)90056-8.

der Simonian R, Charette LJ, McPeek B, Mosteller F: Reporting on methods in clinical trials. New Eng J Med. 1982, 306: 1332-1337.

Detsky AS, Naylor CD, O'Rourke K, McGeer AJ, L'Abbe KA: Incorporating variations in the quality of individual randomized trials into meta-analysis. J Clin Epidemiol. 1992, 45: 255-265. 10.1016/0895-4356(92)90085-2.

Goudas L, Carr DB, Bloch R, Balk E, Ioannidis JPA, Terrin MN: Management of Cancer Pain. Evidence Report/Technology Assessment No. 35 (Contract 290-97-0019 to the New England Medical Center), AHCPR Publication No. 99-E004. Rockville. 2000

Guyatt GH, Sackett DL, Cook DJ: Users' guides to the medical literature. II. How to use an article about therapy or prevention. A. Are the results of the study valid? Evidence-Based Medicine Working Group. J Am Med Assoc. 1993, 270: 2598-2601. 10.1001/jama.270.21.2598.

Khan KS, Ter Riet G, Glanville J, Sowden AJ, Kleijnen J: Undertaking Systematic Reviews of Research on Effectiveness: Centre of Reviews and Dissemination's Guidance for Carrying Out or Commissioning Reviews: York. 2000

McNamara R, Bass EB, Marlene R, Miller J: Management of New Onset Atrial Fibrillation. Evidence Report/Technology Assessment No.12, AHRQ Publication No. 01-E026. Rockville. 2001

Prendiville W, Elbourne D, Chalmers I: The effects of routine oxytocic administration in the management of the third stage of labour: an overview of the evidence from controlled trials. Br J Obstet Gynae Col. 1988, 95: 3-16.

Schulz KF, Chalmers I, Hayes RJ, Altman DG: Empirical evidence of bias. Dimensions of methodological quality associated with estimates of treatment effects in controlled trials. J Am Med Assoc. 1995, 273: 408-412. 10.1001/jama.273.5.408.

The Standards of Reporting Trials Group: A proposal for structured reporting of randomized controlled trials. J Am Med Assoc. 1994, 272: 1926-1931. 10.1001/jama.272.24.1926.

Verhagen AP, de Vet HC, de Bie RA, Kessels AGH, Boers M, Bouter LM, Knipschild PG: The Delphi list: a criteria list for quality assessment of randomized clinical trials for conducting systematic reviews developed by Delphi consensus. J Clin Epidemiol. 1998, 51: 1235-1241. 10.1016/S0895-4356(98)00131-0.

Zaza S, Wright-De Aguero LK, Briss PA, Truman BI, Hopkins DP, Hennessy MH, Sosin DM, Anderson L, Carande-Kullis VG, Teutsch SM, Pappaioanou M: Data collection instrument and procedure for systematic reviews in the guide to community preventive services. Task force on community preventive services. Am J Prevent Med. 2000, 18: 44-74. 10.1016/S0749-3797(99)00122-1.

Haynes BB, Wilczynski N, McKibbon A, Walker CJ, Sinclair J: Developing optimal search strategies for detecting clinically sound studies in MEDLINE. J Am Informatics Assoc. 1994, 1: 447-458.

Greenhalgh T: How to read a paper: papers that report diagnostic or screening tests. BMJ. 1997, 315: 540-543.

Arroll B, Schechter MT, Sheps SB: The assessment of diagnostic tests: a comparison of medical literature in 1982 and 1985. J Gen Int Med. 1988, 3: 443-447.

Lijmer JG, Mol BW, Heisterkamp S, Bonsel GJ, Prins MH, van der Meulen JH, Bossuyt PM: Empirical evidence of design-related bias in studies of diagnostic tests. J Am Med Assoc. 1999, 282: 1061-1066. 10.1001/jama.282.11.1061.

Sheps SB, Schechter MT: The assessment of diagnostic tests. A survey of current medical research. J Am Med Assoc. 1984, 252: 2418-2422. 10.1001/jama.252.17.2418.

McCrory DC, Matchar DB, Bastian L, Dutta S, Hasselblad V, Hickey J, Myers MSE, Nanda K: Evaluation of Cervical Cytology. Evidence Report/Technology Assessment No. 5, AHCPR Publication No.99-E010. Rockville. 1999

Bossuyt PM, Reitsma JB, Bruns DE, Gatsonis CA, Glasziou PP, Irwig LM, Lijmer JG, Moher D, Rennie D, DeVet HCW: Towards complete and accurate reporting of studies of diagnostic accuracy: the STARD initiative. Clin Chem. 2003, 49: 1-6. 10.1373/49.1.1.

Greenhalgh T: How to Read a Paper: Assessing the methodological quality of published papers. BMJ. 1997, 315: 305-308.

Angelillo I, Villari P: Residential exposure to electromagnetic fields and childhood leukaemia: a meta-analysis. Bull World Health Org. 1999, 77: 906-915.

Ariens G, Mechelen W, Bongers P, Bouter L, Van der Wal G: Physical risk factors for neck pain. Scand J Work Environ Health. 2000, 26: 7-19.

Hoogendoorn WE, van Poppel MN, Bongers PM, Koes BW, Bouter LM: Physical load during work and leisure time as risk factors for back pain. Scand J Work Environ Health. 1999, 25: 387-403.

Laupacis A, Wells G, Richardson WS, Tugwell P: Users' guides to the medical literature. V. How to use an article about prognosis. Evidence-Based Medicine Working Group. J Am Med Assoc. 1994, 272: 234-237. 10.1001/jama.272.3.234.

Levine M, Walter S, Lee H, Haines T, Holbrook A, Moyer V: Users' guides to the medical literature. IV. How to use an article about harm. Evidence-Based Medicine Working Group. J Am Med Assoc. 1994, 271: 1615-1619. 10.1001/jama.271.20.1615.

Carey TS, Boden SD: A critical guide to case series reports. Spine. 2003, 28: 1631-1634. 10.1097/00007632-200308010-00001.

Greenhalgh T, Taylor R: How to read a paper: papers that go beyond numbers (qualitative research). BMJ. 1997, 315: 740-743.

Hoddinott P, Pill R: A review of recently published qualitative research in general practice. More methodological questions than answers?. Fam Pract. 1997, 14: 313-319. 10.1093/fampra/14.4.313.

Mays N, Pope C: Quality research in health care: Assessing quality in qualitative research. BMJ. 2000, 320: 50-52. 10.1136/bmj.320.7226.50.

Mays N, Pope C: Rigour and qualitative research. BMJ. 1995, 311: 109-112.

Colditz GA, Miller JN, Mosteller F: How study design affects outcomes in comparisons of therapy. I: Medical. Stats Med. 1989, 8: 441-454.

Turlik MA, Kushner D: Levels of evidence of articles in podiatric medical journals. J Am Pod Med Assoc. 2000, 90: 300-302.

Borghouts JAJ, Koes BW, Bouter LM: The clinical course and prognostic factors of non-specific neck pain: a systematic review. Pain. 1998, 77: 1-13. 10.1016/S0304-3959(98)00058-X.

Spitzer WO, Lawrence V, Dales R, Hill G, Archer MC, Clark P, Abenhaim L, Hardy J, Sampalis J, Pinfold SP, Morgan PP: Links between passive smoking and disease: a best-evidence synthesis. A report of the working group on passive smoking. Clin Invest Med. 1990, 13: 17-46.

Sutton AJ, Abrams KR, Jones DR, Sheldon TA, Song F: Systematic reviews of trials and other studies. Health Tech Assess. 1998, 2: 1-276.

Chestnut RM, Carney N, Maynard H, Patterson P, Mann NC, Helfand M: Rehabilitation for Traumatic Brain Injury. Evidence Report/Technology Assessment No. 2, Agency for Health Care Research and Quality Publication No. 99-E006. Rockville. 1999

Lohr KN, Carey TS: Assessing best evidence: issues in grading the quality of studies for systematic reviews. Joint Commission J Qual Improvement. 1999, 25: 470-479.

Greer N, Mosser G, Logan G, Halaas GW: A practical approach to evidence grading. Joint Commission J Qual Improvement. 2000, 26: 700-712.

Harris RP, Helfand M, Woolf SH, Lohr KN, Mulrow CD, Teutsch SM, Atkins D: Current methods of the U.S. Preventive Services Task Force: a review of the process. Am J Prevent Med. 2001, 20: 21-35. 10.1016/S0749-3797(01)00261-6.

Anonymous: How to read clinical journals: IV. To determine etiology or causation. Can Med Assoc J. 1981, 124: 985-990.

Whitten PS, Mair FS, Haycox A, May CR, Williams TL, Hellmich S: Systematic review of cost effectiveness studies of telemedicine interventions. BMJ. 2002, 324: 1434-1437. 10.1136/bmj.324.7351.1434.

PubMed   PubMed Central   Google Scholar  

Forrest JL, Miller SA: Evidence-based decision making in action: Part 2-evaluating and applying the clinical evidence. J Contemp Dental Pract. 2002, 4: 42-52.

Oxman AD, Guyatt GH: Validation of an index of the quality of review articles. J Clin Epidemiol. 1991, 44: 1271-1278. 10.1016/0895-4356(91)90160-B.

Jones T, Evans D: Conducting a systematic review. Aust Crit Care. 2000, 13: 66-71.

Papadopoulos M, Rheeder P: How to do a systematic literature review. South African J Physiother. 2000, 56: 3-6.

Selker LG: Clinical research in Allied Health. J Allied Health. 1994, 23: 201-228.

Stevens KR: Systematic reviews: the heart of evidence-based practice. AACN Clin Issues. 2001, 12: 529-538.

Devers KJ, Frankel RM: Getting qualitative research published. Ed Health. 2001, 14: 109-117. 10.1080/13576280010021888.

Canadian Journal of Public Health: Review guidelines for qualitative research papers submitted for consideration to the Canadian Journal of Public Health. Can J Pub Health. 2000, 91: I2-

Malterud K: Shared understanding of the qualitative research process: guidelines for the medical researcher. Fam Pract. 1993, 10: 201-206.

Higgs J, Titchen A: Research and knowledge. Physiotherapy. 1998, 84: 72-80.

Maggs-Rapport F: Best research practice: in pursuit of methodological rigour. J Advan Nurs. 2001, 35: 373-383. 10.1046/j.1365-2648.2001.01853.x.

Cutcliffe JR, McKenna HP: Establishing the credibility of qualitative research findings: the plot thickens. J Advan Nurs. 1999, 30: 374-380. 10.1046/j.1365-2648.1999.01090.x.

Andresen EM: Criteria for assessing the tools of disability outcomes research. Arch Phys Med Rehab. 2000, 81: S15-S20. 10.1053/apmr.2000.20619.

Beatie P: Measurement of health outcomes in the clinical setting: applications to physiotherapy. Phys Theory Pract. 2001, 17: 173-185. 10.1080/095939801317077632.

Charnock DF, (Ed): The DISCERN Handbook: Quality criteria for consumer health information on treatment choices. 1998, Radcliffe Medical Press

Pre-publication history

The pre-publication history for this paper can be accessed here: http://www.biomedcentral.com/1471-2288/4/22/prepub

Download references

Author information

Authors and affiliations.

Centre for Allied Health Evidence: A Collaborating Centre of the Joanna Briggs Institute, City East Campus, University of South Australia, North Terrace, Adelaide, 5000, Australia

Persis Katrak, Nicola Massy-Westropp, VS Saravana Kumar & Karen A Grimmer

School of Physiotherapy, The University of Melbourne, Melbourne, 3010, Australia

Andrea E Bialocerkowski

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Karen A Grimmer .

Additional information

Competing interests.

No competing interests.

Authors' contributions

PK Sourced critical appraisal tools

Categorized the content and psychometric properties of critical appraisal tools

AEB Synthesis of findings

Drafted manuscript

NMW Sourced critical appraisal tools

VSK Sourced critical appraisal tools

KAG Study conception and design

Assisted with critiquing critical appraisal tools and categorization of the content and psychometric properties of critical appraisal tools

Drafted and reviewed manuscript

Addressed reviewer's comments and re-submitted the article

Electronic supplementary material

Additional file 1: search strategy. (doc 30 kb), authors’ original submitted files for images.

Below are the links to the authors’ original submitted files for images.

Authors’ original file for figure 1

Authors’ original file for figure 2, authors’ original file for figure 3, authors’ original file for figure 4, authors’ original file for figure 5, rights and permissions.

Reprints and permissions

About this article

Cite this article.

Katrak, P., Bialocerkowski, A.E., Massy-Westropp, N. et al. A systematic review of the content of critical appraisal tools. BMC Med Res Methodol 4 , 22 (2004). https://doi.org/10.1186/1471-2288-4-22

Download citation

Received : 10 May 2004

Accepted : 16 September 2004

Published : 16 September 2004

DOI : https://doi.org/10.1186/1471-2288-4-22

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Research Consumer
• Empirical Basis
• Ally Health Literature
• Critical Appraisal Tool
• Sample Size Justification

BMC Medical Research Methodology

ISSN: 1471-2288

• General enquiries: [email protected]

• Mayo Clinic Libraries
• Systematic Reviews
• Critical Appraisal by Study Design

Systematic Reviews: Critical Appraisal by Study Design

• Knowledge Synthesis Comparison
• Knowledge Synthesis Decision Tree
• Standards & Reporting Results
• Materials in the Mayo Clinic Libraries
• Training Resources
• Review Teams
• Develop & Refine Your Research Question
• Develop a Timeline
• Project Management
• Communication
• PRISMA-P Checklist
• Eligibility Criteria
• Register your Protocol
• Other Resources
• Other Screening Tools
• Grey Literature Searching
• Citation Searching
• Data Extraction Tools
• Minimize Bias
• Synthesis & Meta-Analysis
• Publishing your Systematic Review

Tools for Critical Appraisal of Studies

“The purpose of critical appraisal is to determine the scientific merit of a research report and its applicability to clinical decision making.” 1 Conducting a critical appraisal of a study is imperative to any well executed evidence review, but the process can be time consuming and difficult. 2 The critical appraisal process requires “a methodological approach coupled with the right tools and skills to match these methods is essential for finding meaningful results.” 3 In short, it is a method of differentiating good research from bad research.

Critical Appraisal by Study Design (featured tools)

• Non-RCTs or Observational Studies
• Diagnostic Accuracy
• Animal Studies
• Qualitative Research
• Tool Repository
• AMSTAR 2 The original AMSTAR was developed to assess the risk of bias in systematic reviews that included only randomized controlled trials. AMSTAR 2 was published in 2017 and allows researchers to “identify high quality systematic reviews, including those based on non-randomised studies of healthcare interventions.” 4 more... less... AMSTAR 2 (A MeaSurement Tool to Assess systematic Reviews)
• ROBIS ROBIS is a tool designed specifically to assess the risk of bias in systematic reviews. “The tool is completed in three phases: (1) assess relevance(optional), (2) identify concerns with the review process, and (3) judge risk of bias in the review. Signaling questions are included to help assess specific concerns about potential biases with the review.” 5 more... less... ROBIS (Risk of Bias in Systematic Reviews)
• BMJ Framework for Assessing Systematic Reviews This framework provides a checklist that is used to evaluate the quality of a systematic review.
• CASP Checklist for Systematic Reviews This CASP checklist is not a scoring system, but rather a method of appraising systematic reviews by considering: 1. Are the results of the study valid? 2. What are the results? 3. Will the results help locally? more... less... CASP (Critical Appraisal Skills Programme)
• CEBM Systematic Reviews Critical Appraisal Sheet The CEBM’s critical appraisal sheets are designed to help you appraise the reliability, importance, and applicability of clinical evidence. more... less... CEBM (Centre for Evidence-Based Medicine)
• JBI Critical Appraisal Tools, Checklist for Systematic Reviews JBI Critical Appraisal Tools help you assess the methodological quality of a study and to determine the extent to which study has addressed the possibility of bias in its design, conduct and analysis.
• NHLBI Study Quality Assessment of Systematic Reviews and Meta-Analyses The NHLBI’s quality assessment tools were designed to assist reviewers in focusing on concepts that are key for critical appraisal of the internal validity of a study. more... less... NHLBI (National Heart, Lung, and Blood Institute)
• RoB 2 RoB 2 “provides a framework for assessing the risk of bias in a single estimate of an intervention effect reported from a randomized trial,” rather than the entire trial. 6 more... less... RoB 2 (revised tool to assess Risk of Bias in randomized trials)
• CASP Randomised Controlled Trials Checklist This CASP checklist considers various aspects of an RCT that require critical appraisal: 1. Is the basic study design valid for a randomized controlled trial? 2. Was the study methodologically sound? 3. What are the results? 4. Will the results help locally? more... less... CASP (Critical Appraisal Skills Programme)
• CONSORT Statement The CONSORT checklist includes 25 items to determine the quality of randomized controlled trials. “Critical appraisal of the quality of clinical trials is possible only if the design, conduct, and analysis of RCTs are thoroughly and accurately described in the report.” 7 more... less... CONSORT (Consolidated Standards of Reporting Trials)
• NHLBI Study Quality Assessment of Controlled Intervention Studies The NHLBI’s quality assessment tools were designed to assist reviewers in focusing on concepts that are key for critical appraisal of the internal validity of a study. more... less... NHLBI (National Heart, Lung, and Blood Institute)
• JBI Critical Appraisal Tools Checklist for Randomized Controlled Trials JBI Critical Appraisal Tools help you assess the methodological quality of a study and to determine the extent to which study has addressed the possibility of bias in its design, conduct and analysis.
• ROBINS-I ROBINS-I is a “tool for evaluating risk of bias in estimates of the comparative effectiveness… of interventions from studies that did not use randomization to allocate units… to comparison groups.” 8 more... less... ROBINS-I (Risk Of Bias in Non-randomized Studies – of Interventions)
• NOS This tool is used primarily to evaluate and appraise case-control or cohort studies. more... less... NOS (Newcastle-Ottawa Scale)
• AXIS Cross-sectional studies are frequently used as an evidence base for diagnostic testing, risk factors for disease, and prevalence studies. “The AXIS tool focuses mainly on the presented [study] methods and results.” 9 more... less... AXIS (Appraisal tool for Cross-Sectional Studies)
• NHLBI Study Quality Assessment Tools for Non-Randomized Studies The NHLBI’s quality assessment tools were designed to assist reviewers in focusing on concepts that are key for critical appraisal of the internal validity of a study. • Quality Assessment Tool for Observational Cohort and Cross-Sectional Studies • Quality Assessment of Case-Control Studies • Quality Assessment Tool for Before-After (Pre-Post) Studies With No Control Group • Quality Assessment Tool for Case Series Studies more... less... NHLBI (National Heart, Lung, and Blood Institute)
• Case Series Studies Quality Appraisal Checklist Developed by the Institute of Health Economics (Canada), the checklist is comprised of 20 questions to assess “the robustness of the evidence of uncontrolled, [case series] studies.” 10
• Methodological Quality and Synthesis of Case Series and Case Reports In this paper, Dr. Murad and colleagues “present a framework for appraisal, synthesis and application of evidence derived from case reports and case series.” 11
• MINORS The MINORS instrument contains 12 items and was developed for evaluating the quality of observational or non-randomized studies. 12 This tool may be of particular interest to researchers who would like to critically appraise surgical studies. more... less... MINORS (Methodological Index for Non-Randomized Studies)
• JBI Critical Appraisal Tools for Non-Randomized Trials JBI Critical Appraisal Tools help you assess the methodological quality of a study and to determine the extent to which study has addressed the possibility of bias in its design, conduct and analysis. • Checklist for Analytical Cross Sectional Studies • Checklist for Case Control Studies • Checklist for Case Reports • Checklist for Case Series • Checklist for Cohort Studies
• QUADAS-2 The QUADAS-2 tool “is designed to assess the quality of primary diagnostic accuracy studies… [it] consists of 4 key domains that discuss patient selection, index test, reference standard, and flow of patients through the study and timing of the index tests and reference standard.” 13 more... less... QUADAS-2 (a revised tool for the Quality Assessment of Diagnostic Accuracy Studies)
• JBI Critical Appraisal Tools Checklist for Diagnostic Test Accuracy Studies JBI Critical Appraisal Tools help you assess the methodological quality of a study and to determine the extent to which study has addressed the possibility of bias in its design, conduct and analysis.
• STARD 2015 The authors of the standards note that “[e]ssential elements of [diagnostic accuracy] study methods are often poorly described and sometimes completely omitted, making both critical appraisal and replication difficult, if not impossible.”10 The Standards for the Reporting of Diagnostic Accuracy Studies was developed “to help… improve completeness and transparency in reporting of diagnostic accuracy studies.” 14 more... less... STARD 2015 (Standards for the Reporting of Diagnostic Accuracy Studies)
• CASP Diagnostic Study Checklist This CASP checklist considers various aspects of diagnostic test studies including: 1. Are the results of the study valid? 2. What were the results? 3. Will the results help locally? more... less... CASP (Critical Appraisal Skills Programme)
• CEBM Diagnostic Critical Appraisal Sheet The CEBM’s critical appraisal sheets are designed to help you appraise the reliability, importance, and applicability of clinical evidence. more... less... CEBM (Centre for Evidence-Based Medicine)
• SYRCLE’s RoB “[I]mplementation of [SYRCLE’s RoB tool] will facilitate and improve critical appraisal of evidence from animal studies. This may… enhance the efficiency of translating animal research into clinical practice and increase awareness of the necessity of improving the methodological quality of animal studies.” 15 more... less... SYRCLE’s RoB (SYstematic Review Center for Laboratory animal Experimentation’s Risk of Bias)
• ARRIVE 2.0 “The [ARRIVE 2.0] guidelines are a checklist of information to include in a manuscript to ensure that publications [on in vivo animal studies] contain enough information to add to the knowledge base.” 16 more... less... ARRIVE 2.0 (Animal Research: Reporting of In Vivo Experiments)
• Critical Appraisal of Studies Using Laboratory Animal Models This article provides “an approach to critically appraising papers based on the results of laboratory animal experiments,” and discusses various “bias domains” in the literature that critical appraisal can identify. 17
• CEBM Critical Appraisal of Qualitative Studies Sheet The CEBM’s critical appraisal sheets are designed to help you appraise the reliability, importance and applicability of clinical evidence. more... less... CEBM (Centre for Evidence-Based Medicine)
• CASP Qualitative Studies Checklist This CASP checklist considers various aspects of qualitative research studies including: 1. Are the results of the study valid? 2. What were the results? 3. Will the results help locally? more... less... CASP (Critical Appraisal Skills Programme)
• Quality Assessment and Risk of Bias Tool Repository Created by librarians at Duke University, this extensive listing contains over 100 commonly used risk of bias tools that may be sorted by study type.
• Latitudes Network A library of risk of bias tools for use in evidence syntheses that provides selection help and training videos.

References & Recommended Reading

1.     Kolaski, K., Logan, L. R., & Ioannidis, J. P. (2024). Guidance to best tools and practices for systematic reviews .  British Journal of Pharmacology ,  181 (1), 180-210

2.    Portney LG.  Foundations of clinical research : applications to evidence-based practice.  Fourth edition. ed. Philadelphia: F A Davis; 2020.

3.     Fowkes FG, Fulton PM.  Critical appraisal of published research: introductory guidelines.   BMJ (Clinical research ed).  1991;302(6785):1136-1140.

4.     Singh S.  Critical appraisal skills programme.   Journal of Pharmacology and Pharmacotherapeutics.  2013;4(1):76-77.

5.     Shea BJ, Reeves BC, Wells G, et al.  AMSTAR 2: a critical appraisal tool for systematic reviews that include randomised or non-randomised studies of healthcare interventions, or both.   BMJ (Clinical research ed).  2017;358:j4008.

6.     Whiting P, Savovic J, Higgins JPT, et al.  ROBIS: A new tool to assess risk of bias in systematic reviews was developed.   Journal of clinical epidemiology.  2016;69:225-234.

7.     Sterne JAC, Savovic J, Page MJ, et al.  RoB 2: a revised tool for assessing risk of bias in randomised trials.  BMJ (Clinical research ed).  2019;366:l4898.

8.     Moher D, Hopewell S, Schulz KF, et al.  CONSORT 2010 Explanation and Elaboration: Updated guidelines for reporting parallel group randomised trials.  Journal of clinical epidemiology.  2010;63(8):e1-37.

9.     Sterne JA, Hernan MA, Reeves BC, et al.  ROBINS-I: a tool for assessing risk of bias in non-randomised studies of interventions.  BMJ (Clinical research ed).  2016;355:i4919.

10.     Downes MJ, Brennan ML, Williams HC, Dean RS.  Development of a critical appraisal tool to assess the quality of cross-sectional studies (AXIS).   BMJ open.  2016;6(12):e011458.

11.   Guo B, Moga C, Harstall C, Schopflocher D.  A principal component analysis is conducted for a case series quality appraisal checklist.   Journal of clinical epidemiology.  2016;69:199-207.e192.

12.   Murad MH, Sultan S, Haffar S, Bazerbachi F.  Methodological quality and synthesis of case series and case reports.  BMJ evidence-based medicine.  2018;23(2):60-63.

13.   Slim K, Nini E, Forestier D, Kwiatkowski F, Panis Y, Chipponi J.  Methodological index for non-randomized studies (MINORS): development and validation of a new instrument.   ANZ journal of surgery.  2003;73(9):712-716.

14.   Whiting PF, Rutjes AWS, Westwood ME, et al.  QUADAS-2: a revised tool for the quality assessment of diagnostic accuracy studies.   Annals of internal medicine.  2011;155(8):529-536.

15.   Bossuyt PM, Reitsma JB, Bruns DE, et al.  STARD 2015: an updated list of essential items for reporting diagnostic accuracy studies.   BMJ (Clinical research ed).  2015;351:h5527.

16.   Hooijmans CR, Rovers MM, de Vries RBM, Leenaars M, Ritskes-Hoitinga M, Langendam MW.  SYRCLE's risk of bias tool for animal studies.   BMC medical research methodology.  2014;14:43.

17.   Percie du Sert N, Ahluwalia A, Alam S, et al.  Reporting animal research: Explanation and elaboration for the ARRIVE guidelines 2.0.  PLoS biology.  2020;18(7):e3000411.

18.   O'Connor AM, Sargeant JM.  Critical appraisal of studies using laboratory animal models.   ILAR journal.  2014;55(3):405-417.

• << Previous: Minimize Bias
• Next: GRADE >>
• Last Updated: May 10, 2024 7:59 AM
• URL: https://libraryguides.mayo.edu/systematicreviewprocess

Evidence Synthesis and Systematic Reviews

• Common Review Types
• Other Review Types
• Resources for Reviews by Discipline and Type

Tools for Evidence Synthesis

Evidence synthesis standards, literature searching tools, screening tools, data extraction tools, meta-analysis tools, critical appraisal checklists, grading the strength of evidence tools, risk of bias tools.

• Grey Literature

On this page you will find tools for the different steps of the evidence synthesis process

If you are looking for even more tools check out:

• The Systematic Review Toolbox A community driven directory of systematic review tools.

Checklists, diagrams, reporting guidelines and registration resources to help you plan your evidence synthesis review.

• Open Science Framework (OSF)
• JBI Manual for Evidence Synthesis

Text mining tools to assist in creating a reproducible search.

• PubMed Pub ReMiner
• Yale MeSH Analyzer

Tools used to screen citations generated from searches. These tools will allow for decisions based on inclusion, exclusion, and maybe.

Tools for the process of extracting the relevant information from the studies you have assessed for eligibility in your review.

• Systematic Review Data Repository (SRDR+)
• DistillerSR (Evidence Partners)
• Cochrane Reviews Handbook Chapter on Collecting Data

Tools to perform a meta-analysis and a link to the Biostatistics Consulting Center at Temple University.

• Meta Essentials
• OpenMeta[Analyst]
• Temple University Biostatistics Consulting Center

Checklists used in the process of critically appraising evidence and articles.

• John Hopkins Evidence Based Practice Model

Handbook for grading the quality of evidence and the strength of recommendations using the GRADE approach.

• GRADE Handbook

Risk of bias tool for randomized trials.

• << Previous: Resources for Reviews by Discipline and Type
• Next: Grey Literature >>
• Last Updated: May 16, 2024 11:28 AM
• URL: https://guides.temple.edu/systematicreviews

Temple University

University libraries.

See all library locations

• Library Directory
• Locations and Directions
• Frequently Called Numbers

Need help? Email us at [email protected]

About Systematic Reviews

Choosing the Best Systematic Review Critical Appraisal Tool

Automate every stage of your literature review to produce evidence-based research faster and more accurately.

What is a critical appraisal.

Critical appraisal involves the evaluation of the quality, reliability, and relevance of studies, which is assessed based on quality measures specific to the research question, its related topics, design, methodology, data analysis, and the reporting of different types of systematic reviews .

Planning a critical appraisal starts with identifying or developing checklists. There are several critical appraisal tools that can be used to guide the process, adapting evaluation measures to be relevant to the specific research. It is important to pilot test these checklists and ensure that they are comprehensive enough to tackle all aspects of your systematic review.

What is the Purpose of a Critical Appraisal?

A critical appraisal is an integral part of a systematic review because it helps determine which studies can support the research. Here are some additional reasons why critical appraisals are important.

Assessing Quality

Critical appraisals employ measures specific to the systematic review. Through these, researchers can assess the quality of the studies—their trustworthiness, value, and reliability. This helps weed out substandard reviews, saving researchers’ time that would have been wasted reading full texts.

Determining Relevance

By appraising studies, researchers can determine whether or not they are relevant to the systematic review, such as if they’re connected to the topic or if their results support the research, etc. By doing this, the question “ Can you include a systematic review in a scoping review? ” can also be answered depending on its relevance to the study.

Identifying Flaws

Critical appraisals aim to identify methodological flaws in the literature, helping researchers and readers make informed decisions about the research evidence. They also help reduce the risk of bias when selecting studies.

What to Consider in a Critical Appraisal

Critical appraisals vary as they are specific to the topic, nature, and methodology of each systematic review. However, they generally have the same goal, trying to answer the following questions about the studies being considered:

• Is the study relevant to the research question?
• Is the study valid?
• Did the study use appropriate methods to address the research question?
• Does the study support the findings and evidence claims of the review?
• Are the valid results of the study important?
• Are the valid results of the study applicable to the research?

Learn More About DistillerSR

(Article continues below)

Critical Appraisal Tools

There are hundreds of tools and worksheets that can serve as a guide through the critical appraisal process. Here are just some of the most common ones to consider:

• AMSTAR – to examine the effectiveness of interventions.
• CASP – to appraise randomized control trials, systematic reviews, cohort studies, case-control studies, qualitative research, economic evaluations, diagnostic tests, and clinical prediction rules.
• Cochrane Risk of Bias Tool – to assess the risk of bias of randomized control trials (RCTs).
• GRADE – to grade the quality of evidence in healthcare research and policy.
• JBI Critical Tools – to assess trustworthiness, relevance, and results of published papers.
• NOS – to assess the quality of non-randomized studies in meta-analyses.
• ROBIS – to assess the risk of bias in interventions, diagnosis, prognosis, and etiology.
• STROBE – to address cohort, case-control, and conduct cross-sectional studies.

What is the Best Critical Appraisal Tool?

There is no single best critical appraisal tool for any study design, nor is there a generic one that can be expected to consistently do well when used across different study types.

Critical appraisal tools vary considerably in intent, components, and construction, and the right one for your systematic review is the one that addresses the components that you need to tackle and ensures that your research results in comprehensive, unbiased, and valid findings.

3 Reasons to Connect

Library Services

UCL LIBRARY SERVICES

• Guides and databases
• Library skills
• Systematic reviews

Describing and appraising studies

• What are systematic reviews?
• Types of systematic reviews
• Formulating a research question
• Identifying studies
• Searching databases
• Synthesis and systematic maps
• Software for systematic reviews
• Online training and support
• Live and face to face training
• Individual support
• Further help

On this page:

• Describing stuides

Examples of describing studies

Critical appraisal of quality and relevance, appraising the quality of a systematic review, describing studies.

Studies that are used in a review are described in a standardised way that is suitable for each review.  The detail provided facilitates transparency in how each study contributes to the overall findings of the review, and the overall reliabillity of the review.

There are three key reasons for describing (or coding) the studies in a systematic review.

• To know more about the included studies. Studies may be described on characteristics such as the aims, methods, or particular elements to describe the research sample and any outcomes measured.  Historically such descriptions about the studies may have been limited to basic information such as authors names, place of publication, and research methods. It is now recognized that information about what has or has not been studied is a useful product in its own right, as highlighted in the box on systematic evidence maps .
• To identify the research findings of the individual studies to be synthesized
• To identify the methods used in each study, so the study can be critically appraised for trustworthiness and relevance to the review. Methodological aspects of each study might be described in terms of how sampling was undertaken, recruitment of the sample, data collection methods, data analysis methods.
• Coding for systematic reviews and meta-analysis, Sandra Wilson Video (78 mins) on describing studies)

• Young people, pregnancy and social exclusion Brunton et al 2006. An example of the coding used for all the included studies for a review on Young people, pregnancy and social exclusion.

The EPPI-Centre coding guidelines in education:

• Education guidelines version 0.94 (2001) Guideline for descriptive keywording.
• Education guidelines version 0.97 (2003) Guideline for extracting and appraising data from studies.

Critical appraisal involves checking the quality, reliability and relevance of the studies in the review in relation to the review question. It appraises each study in terms of the following aspects:. 

• Is the study relevant to the research question?
• Is the study valid? E.g. Were the study methods applied appropriately?
• Were appropriate methods used in relation to the review question?

In addition, the studies are collectively appraised in terms of how they support the review findings and evidence claims of the review. For example, if the research evidence comprises of studies that have wide variation of findings, this reduces the strength of the evidence claims.

There are many standardised tools available for critical appraisal depending on the study design and the type of review. The approach to critical appraisal and the appraisal decisions for each study should be reported.

• Critical Appraisal Skills Programme (CASP) Tools (checklists) for systematic reviews, randomised controlled trials, cohort studies, case control studies, economic evaluations, diagnostic studies, qualitative studies, clinical prediction rule
• EBP checklists: University of Glasgow Adaptations of the CASP checklists for appraising other types of study, including an education intervention, and studies on treatment and prevention, and guidelines.
• Newcastle-Ottawa Scale Useful for assessing quality of non-randomised studies such as case-control and cohort studies.
• Risk of Bias Cochrane tool Version 2 of the Cochrane risk-of-bias tool for randomized trials (RoB 2). RoB 2 is structured into a fixed set of domains of bias, focussing on different aspects of trial design, conduct, and reporting.

Commonly-used tools for appraising research evidence in reviews:

• GRADE For appraising reviews of effectiveness.
• CerQUAL For appraising reviews including qualitative research evidence.

It is important for users of systematic reviews to consider the quality of the whole review. There are three separate elements that contribute an appraisal:

• the quality and relevance of the methods used to address the review questions;
• the quality and relevance  of the methods used by the individual studies included in the review;
• the nature and extent of the total evidence from studies included in the review.

There are tools to help with the appraisal of a whole review. Some of these are specific to certain types of reviews, and others are more generic. 

• JBI checklist for systematic reviews An appraisal tool that can be used for a range of types of systematic reviews. From the Joanna Briggs Institute.

Some tools focus only on appraising the methods of specific types of reviews:

• AMSTAR 2 Used to appraise only reviews that examine the effectiveness of interventions.
• ROBIS A tool from the University of Bristol that focuses on testing statistical bias in certain types of reviews.

Further reading:

• Evidence Standards: A Dimensions of Difference Framework for Justifiable Evidence Claim Gough (2016). This short paper provides an overarching conceptual framework for considering the main dimensions involved in appraising evidence claims of a review.

Related guides

• LibrarySkills@UCL: Evaluating information
• << Previous: Searching databases
• Next: Synthesis and systematic maps >>
• Last Updated: Apr 4, 2024 10:09 AM
• URL: https://library-guides.ucl.ac.uk/systematic-reviews

Cookies on this website

We use cookies to ensure that we give you the best experience on our website. If you click 'Accept all cookies' we'll assume that you are happy to receive all cookies and you won't see this message again. If you click 'Reject all non-essential cookies' only necessary cookies providing core functionality such as security, network management, and accessibility will be enabled. Click 'Find out more' for information on how to change your cookie settings.

Critical Appraisal tools

Critical appraisal worksheets to help you appraise the reliability, importance and applicability of clinical evidence.

Critical appraisal is the systematic evaluation of clinical research papers in order to establish:

• Does this study address a  clearly focused question ?
• Did the study use valid methods to address this question?
• Are the valid results of this study important?
• Are these valid, important results applicable to my patient or population?

If the answer to any of these questions is “no”, you can save yourself the trouble of reading the rest of it.

This section contains useful tools and downloads for the critical appraisal of different types of medical evidence. Example appraisal sheets are provided together with several helpful examples.

Critical Appraisal Worksheets

• Systematic Reviews  Critical Appraisal Sheet
• Diagnostics  Critical Appraisal Sheet
• Prognosis  Critical Appraisal Sheet
• Randomised Controlled Trials  (RCT) Critical Appraisal Sheet
• Critical Appraisal of Qualitative Studies  Sheet
• IPD Review  Sheet

Chinese - translated by Chung-Han Yang and Shih-Chieh Shao

• Systematic Reviews  Critical Appraisal Sheet
• Diagnostic Study  Critical Appraisal Sheet
• Prognostic Critical Appraisal Sheet
• RCT  Critical Appraisal Sheet
• IPD reviews Critical Appraisal Sheet
• Qualitative Studies Critical Appraisal Sheet 

German - translated by Johannes Pohl and Martin Sadilek

• Systematic Review  Critical Appraisal Sheet
• Diagnosis Critical Appraisal Sheet
• Prognosis Critical Appraisal Sheet
• Therapy / RCT Critical Appraisal Sheet

Lithuanian - translated by Tumas Beinortas

• Systematic review appraisal Lithuanian (PDF)
• Diagnostic accuracy appraisal Lithuanian  (PDF)
• Prognostic study appraisal Lithuanian  (PDF)
• RCT appraisal sheets Lithuanian  (PDF)

Portugese - translated by Enderson Miranda, Rachel Riera and Luis Eduardo Fontes

• Portuguese – Systematic Review Study Appraisal Worksheet
• Portuguese – Diagnostic Study Appraisal Worksheet
• Portuguese – Prognostic Study Appraisal Worksheet
• Portuguese – RCT Study Appraisal Worksheet
• Portuguese – Systematic Review Evaluation of Individual Participant Data Worksheet
• Portuguese – Qualitative Studies Evaluation Worksheet

Spanish - translated by Ana Cristina Castro

• Systematic Review  (PDF)
• Diagnosis  (PDF)
• Prognosis  Spanish Translation (PDF)
• Therapy / RCT  Spanish Translation (PDF)

Persian - translated by Ahmad Sofi Mahmudi

• Prognosis  (PDF)
• PICO  Critical Appraisal Sheet (PDF)
• PICO Critical Appraisal Sheet (MS-Word)
• Educational Prescription  Critical Appraisal Sheet (PDF)

Explanations & Examples

• Pre-test probability
• SpPin and SnNout
• Likelihood Ratios

Systematic Reviews

• Introduction
• About Meta-Analysis
• Planning a Review
• Other Guidelines
• Protocol Registration
• Hedges & Literature Searching
• Grey Literature This link opens in a new window

Appraisal/Risk of Bias

Levels of evidence, other appraisal tools, cochrane critical appraisal training videos.

• For Librarians

• AMSTAR (A MeaSurement Tool to Assess systematic Reviews) 2: A critical appraisal tool for systematic reviews that include randomised or non-randomised studies of healthcare interventions, or both. Shea BJ et al. BMJ. 2017 Sep 21;358:j4008. Instrument for critically appraising systematic reviews of randomized controlled clinical trials.
• AMSTAR Checklist
• AMSTAR Instructions
• CASP: Critical Appraisal Skills Program Checklists "This set of eight critical appraisal tools are designed to be used when reading research, these include tools for Systematic Reviews, Randomised Controlled Trials, Cohort Studies, Case Control Studies, Economic Evaluations, Diagnostic Studies, Qualitative studies and Clinical Prediction Rule."
• Centre for Evidence-Based Medicine (CEBM) "Critical appraisal worksheets to help appraise the reliability, importance and applicability of clinical evidence." Worksheets for SRs, Diagnostics, Prognosis, RCTs, Qualitative and Individual Patient Data (IPD).
• Cochrane risk-of-bias tool for randomized trials (ROB 2) RoB 2 is structured into a fixed set of domains of bias, focussing on different aspects of trial design, conduct, and reporting.
• JBI Critical Appraisal Tools "JBI’s critical appraisal tools assist in assessing the trustworthiness, relevance and results of published papers."
• Newcastle-Ottawa Quality Assessment Scale for Cohort Studies
• Newcastle-Ottawa Quality Assessment Scale for Case Control Studies
• Newcastle-Ottawa Scale for assessing the quality of nonrandomised studies in meta-analyses
• ROBINS-I Risk Of Bias In Non-randomised Studies - of Interventions (ROBINS-I)
• CEBM Levels of Evidence
• GRADE: an emerging consensus on rating quality of evidence and strength of recommendations. Guyatt, G. H., Oxman, A. D., Vist, G. E., Kunz, R., Falck-Ytter, Y., Alonso-Coello, P., Schünemann, H. J., & GRADE Working Group (2008). BMJ (Clinical research ed.), 336(7650), 924–926.
• Johns Hopkins Evidence-Based Practice for Nurses and Healthcare Professionals Model Research Evidence Appraisal Tool (2022) Appendix E: Research Evidence Appraisal Tool
• JBI Grades of Recommendation "Grades of Recommendation are used to assist healthcare professionals when implementing evidence into practice."
• JBI Levels of Evidence "These levels are intended to be used alongside the supporting document outlining their use."
• JBI Manual for Evidence Synthesis
• Mixed-Methods Appraisal Tool (MMAT) "The MMAT is a critical appraisal tool that is designed for the appraisal stage of systematic mixed studies reviews, i.e., reviews that include qualitative, quantitative and mixed methods studies. It permits to appraise the methodological quality of five categories to studies: qualitative research, randomized controlled trials, non-randomized studies, quantitative descriptive studies, and mixed methods studies."
• Introduction to Critical Appraisal
• Systematic Reviews & Meta Analysis
• Randomised Controlled Trials
• Cohort Studies
• Case Control Studies
• Cross Sectional Studies
• Diagnostic Studies

• << Previous: Grey Literature
• Next: Miscellany >>
• Last Updated: May 14, 2024 12:53 PM
• URL: https://guides.libraries.emory.edu/SRs

CASP Checklists

How to use our CASP Checklists

Referencing and Creative Commons

• Online Training Courses
• CASP Workshops
• What is Critical Appraisal
• Study Designs
• Useful Links
• Bibliography
• View all Tools and Resources
• Testimonials

Critical Appraisal Checklists

We offer a number of free downloadable checklists to help you more easily and accurately perform critical appraisal across a number of different study types.

The CASP checklists are easy to understand but in case you need any further guidance on how they are structured, take a look at our guide on how to use our CASP checklists .

CASP Checklist: Systematic Reviews with Meta-Analysis of Observational Studies

CASP Checklist: Systematic Reviews with Meta-Analysis of Randomised Controlled Trials (RCTs)

CASP Randomised Controlled Trial Checklist

• Print & Fill

CASP Systematic Review Checklist

CASP Qualitative Studies Checklist

CASP Cohort Study Checklist

CASP Diagnostic Study Checklist

CASP Case Control Study Checklist

CASP Economic Evaluation Checklist

CASP Clinical Prediction Rule Checklist

Checklist Archive

• CASP Randomised Controlled Trial Checklist 2018 fillable form
• CASP Randomised Controlled Trial Checklist 2018

CASP Checklist

Need more information?

• Online Learning
• Privacy Policy

Critical Appraisal Skills Programme

Critical Appraisal Skills Programme (CASP) will use the information you provide on this form to be in touch with you and to provide updates and marketing. Please let us know all the ways you would like to hear from us:

We use Mailchimp as our marketing platform. By clicking below to subscribe, you acknowledge that your information will be transferred to Mailchimp for processing. Learn more about Mailchimp's privacy practices here.

Copyright 2024 CASP UK - OAP Ltd. All rights reserved Website by Beyond Your Brand

• Methodology
• Open access
• Published: 26 March 2019

SANRA—a scale for the quality assessment of narrative review articles

• Christopher Baethge   ORCID: orcid.org/0000-0001-6246-3674 1 , 2 ,
• Sandra Goldbeck-Wood 1 , 3 &
• Stephan Mertens 1  

Research Integrity and Peer Review volume  4 , Article number:  5 ( 2019 ) Cite this article

139k Accesses

641 Citations

38 Altmetric

Metrics details

Narrative reviews are the commonest type of articles in the medical literature. However, unlike systematic reviews and randomized controlled trials (RCT) articles, for which formal instruments exist to evaluate quality, there is currently no instrument available to assess the quality of narrative reviews. In response to this gap, we developed SANRA, the Scale for the Assessment of Narrative Review Articles.

A team of three experienced journal editors modified or deleted items in an earlier SANRA version based on face validity, item-total correlations, and reliability scores from previous tests. We deleted an item which addressed a manuscript’s writing and accessibility due to poor inter-rater reliability. The six items which form the revised scale are rated from 0 (low standard) to 2 (high standard) and cover the following topics: explanation of (1) the importance and (2) the aims of the review, (3) literature search and (4) referencing and presentation of (5) evidence level and (6) relevant endpoint data. For all items, we developed anchor definitions and examples to guide users in filling out the form. The revised scale was tested by the same editors (blinded to each other’s ratings) in a group of 30 consecutive non-systematic review manuscripts submitted to a general medical journal.

Raters confirmed that completing the scale is feasible in everyday editorial work. The mean sum score across all 30 manuscripts was 6.0 out of 12 possible points (SD 2.6, range 1–12). Corrected item-total correlations ranged from 0.33 (item 3) to 0.58 (item 6), and Cronbach’s alpha was 0.68 (internal consistency). The intra-class correlation coefficient (average measure) was 0.77 [95% CI 0.57, 0.88] (inter-rater reliability). Raters often disagreed on items 1 and 4.

Conclusions

SANRA’s feasibility, inter-rater reliability, homogeneity of items, and internal consistency are sufficient for a scale of six items. Further field testing, particularly of validity, is desirable. We recommend rater training based on the “explanations and instructions” document provided with SANRA. In editorial decision-making, SANRA may complement journal-specific evaluation of manuscripts—pertaining to, e.g., audience, originality or difficulty—and may contribute to improving the standard of non-systematic reviews.

Peer Review reports

Narrative review articles are common in the medical literature. Bastian et al. found that they constitute the largest share of all text types in medicine and they concluded that they “remain the staple of medical literature” [ 1 ]. Narrative reviews also appear popular among both authors and readers, and it is plausible to assume that they exercise an enormous influence among doctors in clinical practice and research. However, because their quality varies widely, they have frequently been compared in blanket, negative terms with systematic reviews.

We use the term narrative review to refer to an attempt to summarize the literature in a way which is not explicitly systematic, where the minimum requirement for the term systematic relates to the method of the literature search, but in a wider sense includes a specific research question and a comprehensive summary of all studies [ 2 ].

While systematic reviews are not per se superior articles and while certain systematic reviews have been criticized lately [ 3 ], non-systematic reviews or narrative reviews have been widely criticized as unreliable [ 1 , 4 ]. Hence, the hierarchy of evidence-based medicine places systematic reviews much higher than non-systematic ones. However, it is likely—and even desirable—that good quality narrative reviews will continue to play an important role in medicine: while systematic reviews are superior to narrative reviews in answering specific questions (for example, whether it is advisable to switch an antidepressant among antidepressant non-responders in patients with major depressive disorder [ 5 ]), narrative reviews are better suited to addressing a topic in wider ways (for example, outlining the general principles of diagnosing and treating depression [ 6 ]).

Critical appraisal tools have been developed for systematic reviews (e.g., AMSTAR 2 [A MeaSurement Tool to Assess Systematic Reviews] [ 7 ]) and papers on RCTs (e.g., the CASP [Critical Appraisal Skills Program] checklist for randomized trials [ 8 ]) and other types of medical studies. For narrative reviews, in contrast, no critical appraisal, or quality assessment tool is available. Such a tool, however, if simple and brief enough for day-to-day use, may support editors in choosing or improving manuscripts, help reviewers and readers in assessing the quality of a paper, and aid authors in preparing narrative reviews. It may improve the general quality of narrative reviews.

As a consequence, we have developed SANRA, the Scale for the Assessment of Narrative Review Articles, a brief critical appraisal tool for the assessment of non-systematic articles. Here, we present the revised scale and the results of a field test regarding its feasibility, item-total correlation, internal consistency, reliability, and criterion validity.

SANRA was developed between 2010 and 2017 by three experienced editors (CB, SGW, and SM) working at a general medical journal, Deutsches Ärzteblatt , the journal of the German Medical Association and the National Association of Statutory Health Insurance Physicians . It is intended to be a simple and brief quality assessment instrument not only to assist editors in their decisions about manuscripts, but also to help reviewers and readers in their assessment of papers and authors in writing narrative reviews.

Two earlier, seven-item versions of SANRA have been developed and tested by the authors, the first in 10 narrative reviews from the field of neurology as retrieved through a PubMed search, the second among 12 consecutive narrative reviews submitted to Deutsches Ärzteblatt —both showing satisfactory internal consistency and inter-rater reliability [ 9 ].

The current version of SANRA [ 10 ] has been revised by the authors in 2014 in order to simplify the scale and make it more robust. We simplified the wording of the items, and we deleted an item addressing a manuscript’s writing and accessibility because ratings of that item differed considerably. The six items that form the revised scale are rated in integers from 0 (low standard) to 2 (high standard), with 1 as an intermediate score. The maximal sum score is 12.

The sum score of the scale is intended to measure the construct “quality of a narrative review article” and covers the following topics: explanation of the review’s importance (item 1) and statement of the aims (item 2) of the review, description of the literature search (item 3), referencing (item 4), scientific reasoning (item 5), and presentation of relevant and appropriate endpoint data (item 6) (Fig.  1 ). For all items, we developed anchor definitions and examples to guide users in filling out the instrument, provided in the document “explanations and instructions,” accompanying the scale. This document was also edited to improve clarity (Fig.  2 ).

SANRA - Scale

SANRA—explanations and instructions document

In 2015, one rater (CB) screened all submissions to Deutsches Ärzteblatt in 2015, and the first 30 consecutive review manuscripts without systematic literature searches were selected for inclusion in the present study. All three raters (CB, SGW, and SM) are editors, with, in 2015, at least 10 years of experience each. They scored the manuscripts independently and blinded to each other’s ratings.

Statistical analysis

Descriptive data are shown as means or medians, as appropriate, and as ranges, standard deviations, or confidence intervals. This study aimed at testing SANRA’s internal consistency (Cronbach’s alpha) and the item-total correlation—indicating whether the items measure the same phenomenon, here different aspects of review paper quality—as well as SANRA’s inter-rater reliability with regard to its sum score. Inter-rater reliability, as a measure of the consistency among different raters, was expressed as the average measure intra-class correlation, ICC, using a two-way random effects model (consistency definition). As an approximation of SANRA’s criterion validity (Is the score predictive of other indicators of paper quality, e.g., acceptance and rejection or citations?), we analyzed post hoc whether average sum scores of SANRA were associated with the decision to accept or reject the 30 manuscripts under study (point biserial correlation for the association between a dichotomous and a continuous variable). All calculations were carried out using SPSS. Where possible, the presentation follows the recommendations of the Guidelines for Reporting Reliability and Agreement Studies (GRRAS) [ 11 ].

All 90 ratings (3 raters × 30 manuscripts) were used for statistical analysis. The mean sum score across all 30 manuscripts ( N  = 90) was 6.0 out of 12 possible points (SD 2.6, range 1–12, median 6). Highest scores were rated for item 4 (mean 1.25; SD 0.70), item 2 (mean 1.14; SD 0.84), and item 1 (mean 1.1; SD 0.69) whereas items 6, 5, and 3 had the lowest scores (means of 0.81 (SD 0.65), 0.83 (SD 0.67), and 0.84 (SD 0.60), respectively) (all single-item medians: 1).

The scale’s internal consistency, measured as Cronbach’s alpha, was 0.68. Corrected item-total correlations ranged from 0.33 to 0.58 (Table  1 ). Tentative deletions of each item to assess the effect of these on consistency showed reduced internal consistency with every deleted item (0.58–0.67) (as shown by the alpha values in Table  1 ).

Across 180 single-item ratings (6 items × 30 manuscripts), the maximum difference among the 3 raters was 2 in 12.8% ( n  = 23; most often in items 1, 2, and 4), in 56.7% ( n  = 102), the raters differed by no more than 1 point, and in 30.6% ( n  = 55), they entirely agreed (most often in items 2 and 3). The intra-class correlation coefficient (average measure) amounted to 0.77 [95% CI 0.57, 0.88; F 4.3; df 29, 58]. Disagreements most often occurred with regard to items 1 and 4.

Average SANRA sum scores of the 30 manuscripts were modestly associated with the editorial decision of acceptance (mean score 6.6, SD 1.9; n  = 17) or rejection (mean score 5.1, SD 2.1; n  = 13): point biserial correlation of 0.37 ( t  = 2.09, df 28; two-sided p  = 0.046).

All raters confirmed that completing the scale is feasible in everyday editorial work.

This study yielded three important findings: (1) SANRA can be applied to manuscripts in everyday editorial work. (2) SANRA’s internal consistency and item-total correlation are sufficient. (3) SANRA’s inter-rater reliability is satisfactory.

Feasibility

It is our experience with the current and earlier SANRA versions that editors, once accustomed to the scale, can integrate the scale into their everyday routine. It is important, however, to learn how to fill out SANRA. To this end, together with SANRA, we provide definitions and examples in the explanations and instructions document, and we recommend that new users train filling out SANRA using this resource. Editorial teams or teams of scientists and/or clinicians may prefer to learn using SANRA in group sessions.

Consistency and homogeneity

With Cronbach’s alpha of 0.68 and corrected item-total correlations between 0.33 and 0.58, we consider the scale’s consistency and item homogeneity sufficient for widespread application. It should be noted that because coefficient alpha increases with the number of items [ 12 ], simplifying a scale by reducing the number of items—as we did—may decrease internal consistency. However, this needs to be balanced against the practical need for brevity. In fact, the earlier seven-item versions of SANRA had higher values of alpha: 0.80 and 0.84, respectively [ 9 ]. Still, the number of items is not necessarily the only explanation for differences in alpha values. For example, the manuscripts included in the two earlier studies may have been easier to rate.

Inter-rater reliability

The scale’s intra-class correlation (0.77 after 0.76 in [ 9 ]) indicates that SANRA can be used reliably by different raters—an important property of a scale that may be applied for manuscript preparation and review, in editorial decision-making, or even in research on narrative reviews. Like internal consistency, reliability increases with the number of items [ 12 ], and there is a trade-off between simplicity (e.g., a small number of items) and reliability. While the ICC suggests sufficient reliability, however, the lower confidence limit (0.57) does not preclude a level of reliability normally deemed unacceptable in most applications of critical appraisal tools. This finding underscores the importance of rater training. Raters more often disagreed on items 1 and 4. After the study, we have therefore slightly edited these items, along with items 5 and 6 which we edited for clarity. In the same vein, we revised our explanations and instructions document.

It is important to bear in mind that testing of a scale always relates only to the setting of a given study. Thus, in the strict sense, the results presented here are not a general feature of SANRA but of SANRA filled out by certain raters with regard to a particular sample of manuscripts. However, from our experience, we trust that our setting is similar to that of many journals, and our sample of manuscripts represents an average group of papers. As a consequence, we are confident SANRA can be applied by other editors, reviewers, readers, and authors.

In a post hoc analysis, we found a modest, but statistically significant correlation of SANRA sum scores with manuscript acceptance. We interpret this as a sign of criterion validity, but emphasize that this is both a post hoc result and only a weak correlation. The latter, however, points to the fact that, at the level of submitted papers, other aspects than quality alone influence editorial decision-making: for example, whether the topic has been covered in the journal recently or whether editors believe that authors or topics of manuscripts have potential, even with low initial SANRA scores. SANRA will therefore often be used as one, and not the only, decision aid. Also, the decision to accept a paper has been made after the papers had been revised.

Moreover, additional results on criterion validity are needed, as are results on SANRA’s construct validity. On the other hand, SANRA’s content validity, defined as a scale’s ability to completely cover all aspects of a construct, will be restricted because we decided to limit the scale to six items, too few to encompass all facets of review article quality—SANRA is a critical appraisal tool and not a reporting guideline. For example, we deleted an item on the accessibility of the manuscript. Other possible domains that are not part of SANRA are, for example, originality of the manuscript or quality of tables and figures. These features are important, but we believe the six items forming SANRA are a core set that sufficiently indicates the quality of a review manuscript and, at the same time, is short enough to be applied without too much time and effort. SANRA’s brevity is also in contrast to other tools to assess articles, such as AMSTAR 2, for systematic reviews, or, to a lesser extent, CASP for RCTs, with its 16 and 11 items, respectively.

Throughout this paper we have referred to the current version of SANRA as the revision of earlier forms. This is technically true. However, because it is normal that scales go through different versions before publication and because this paper is first widespread publication of SANRA, we propose to call the present version simpy SANRA.

While medicine has achieved a great deal in the formalization and improvement of the presentation of randomized trials and systematic review articles, and also a number of other text types in medicine, much less work have been done with regard to the most frequent form of medical publications, the narrative review. There are exceptions: Gasparyan et al. [ 13 ], for example, have provided guidance for writing narrative reviews, and Byrne [ 14 ] as well as Pautasso [ 15 ] has written, from different angles, thoughtful editorials on improving narrative reviews and presented lists of key features of writing a good review—lists that naturally overlap with SANRA items (e.g., on referencing). These lists, however, are not tested scales and not intended for comparing different manuscripts. SANRA can be used in comparisons of manuscripts the way we used it in our editorial office, that is, in one setting. At the present time, however, it seems unwise to compare manuscripts across different settings because, so far, there are no established cut-offs for different grades of quality (e.g., poor-fair-moderate-good-very good). Still, in our experience, a score of 4 or below indicates very poor quality.

Limitations

The main limitation of this study is its sample size. While, in our experience, a group of 30 is not unusual in testing scales, it represents a compromise between the aims of representativeness for our journal and adequate power and feasibility; it took us about 6 months to sample 30 consecutive narrative reviews. Also, in this study, the authors of the scale were also the test-raters, and it is possible that inter-rater reliability is lower in groups less familiar with the scale. As for most scales, this underscores the importance of using the instructions that belong to the scale, in the present case the explanations and instructions document. It is also advisable to train using the scale before applying SANRA for manuscript rating. In addition, by design, this is not a study of test-retest reliability, another important feature of a scale. Finally, as previously acknowledged, although we believe in the representativeness of our setting for medical journals, the present results refer to the setting of this study, and consistency and reliability measures are study-specific.

We present SANRA, a brief scale for the quality assessment of narrative review articles, the most widespread form of article in the medical literature. We suggest SANRA can be integrated into the work of editors, reviewers, and authors. We encourage readers to consider using SANRA as an aid to critically appraising articles, and authors to consider its use on preparing narrative reviews, with a view to improving the quality of submitted and published manuscripts.

SANRA and its explanations and instructions document are available (open access) at: https://www.aerzteblatt.de/down.asp?id=22862 , https://www.aerzteblatt.de/down.asp?id=22861 .

Abbreviations

A MeaSurement Tool to Assess Systematic Reviews

Critical Appraisal Skills Program

Guidelines for Reporting Reliability and Agreement Studies

Intra-class correlation

Randomized controlled trial

Scale for the Assessment of Narrative Review Articles

Bastian H, Glasziou P, Chalmers I. Seventy-five trials and eleven systematic reviews a day: how will we ever keep up? PLoS Med. 2010;7(9):e1000326. https://doi.org/10.1371/journal.pmed.1000326 .

Article   Google Scholar  

Higgins JPT, Green S. (eds.). Cochrane Handbook for Systematic Reviews and Interventions. Version 5.1.0. The Cochrane Collaboration 2011, section 1.2.2., www.handbook.cochrane.org , retrieved on Oct 31, 2018.

Ioannidis JP. The mass production of redundant, misleading, and conflicted systematic reviews and meta-analyses. Milbank Q. 2016;94(3):485–514. https://doi.org/10.1111/1468-0009.12210 .

Mulrow CD. The medical review article: state of the science. Ann Intern Med. 1987;106:485–8.

Bschor T, Kern H, Henssler J, Baethge C. Switching the antidepressant after nonresponse in adults with major depression: a systematic literature search and meta-analysis. J Clin Psychiatry. 2018;79(1):16r10749. https://doi.org/10.4088/JCP.16r10749 .

Bschor T, Adli M. Treatment of depressive disorders. Dtsch Arztebl Int. 2008;105(45):782–92.

Google Scholar  

Shea BJ, Reeves BC, Wells G, Thuku M, Hamel C, Moran J, Moher D, Tugwell P, Welch V, Kristjansson E, Henry DA. AMSTAR 2: a critical appraisal tool for systematic reviews that include randomised or non-randomised studies of healthcare interventions, or both. BMJ. 2017;358:J4008.

Critical Appraisal Skills Programme 2018. CASP Randomised controlled trial checklist available at: https://casp-uk.net/casp-tools-checklists/ . Accessed: 31 Oct 2018.

Baethge C, Mertens S, Goldbeck-Wood S. Development of a quality score for the assessment of non-systematic review articles (SANRA). In: Poster, Seventh International Congress on Peer Review and Biomedical Publication. USA: Chicago, Illinois; 2013.

Baethge C, Goldbeck-Wood S, Mertens S. A scale for the assessment of non-systematic review articles (SANRA). In: Poster, Eighth International Congress on Peer and Biomedical Publication. USA: Chicago, Illinois; 2017.

Kottner J, Audige L, Brorson S, Donner A, Gajewski BJ, Hrobjartsson A, Roberts C, Shoukri M, Streiner DL. Guidelines for reporting reliability and agreement studies (GRASS) were proposed. J Clin Epidemiol. 2011;64:96–106.

Streiner DL, Norman GR. Health measurment scales—a practical guide to their development and use. Fourth edition. New York: Oxford University Press; 2008.

Book   Google Scholar  

Gasparyan AY, Ayvazyan L, Blackmore H, Kitas GD. Writing a narrative biomedical review: considerations for authors, peer reviewers, and editors. Rheumatol Int. 2011;31:1409–17.

Byrne JA. Improving the peer review of narrative literature reviews. Research Integrity and Peer Review. 2016;1(12). https://doi.org/10.1186/s41073-016-0019-2 .

Pautasso M. Ten simple rules for writing a literature review. PLoS Comput Biol. 2013;9(7):e1003149. https://doi.org/10.1371/journal.pcbi.1003149 .

Download references

Acknowledgements

This work has been presented at the Eighth International Congress on Peer Review and Scientific Publication in Chicago, Illinois, USA. (September 10-12, 2017) and at the 14th EASE Conference in Bucharest, Romania (June 8-10, 2018).

This work has not been externally funded.

Availability of data and materials

The dataset generated during the course of this study is available from the authors upon request.

Author information

Authors and affiliations.

Deutsches Ärzteblatt and Deutsches Ärzteblatt International, Dieselstraße 2, D-50859, Cologne, Germany

Christopher Baethge, Sandra Goldbeck-Wood & Stephan Mertens

Department of Psychiatry and Psychotherapy, University of Cologne Medical School, Cologne, Germany

Christopher Baethge

BMJ Sexual and Reproductive Health, London, UK

Sandra Goldbeck-Wood

You can also search for this author in PubMed   Google Scholar

Contributions

All authors (CB, SM, and SGW) made substantial contributions to the conception of the study and to the acquisition and interpretation of data. CB analyzed the data and drafted the manuscript. SM and SGW revised the draft critically for important intellectual content. All authors sufficiently participated in this work to take public responsibility for its content, all finally approved the manuscript, and all are accountable for every aspect of this project.

Corresponding author

Correspondence to Christopher Baethge .

Ethics declarations

Ethics approval and consent to participate.

Not applicable

Consent for publication

Competing interests.

Non-financial competing interest: all authors (CB, SM, and SGW) had their part in the development of the scale under study. The authors declare that they have no financial competing interests.

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License ( http://creativecommons.org/licenses/by/4.0/ ), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated.

Reprints and permissions

About this article

Cite this article.

Baethge, C., Goldbeck-Wood, S. & Mertens, S. SANRA—a scale for the quality assessment of narrative review articles. Res Integr Peer Rev 4 , 5 (2019). https://doi.org/10.1186/s41073-019-0064-8

Download citation

Received : 02 December 2018

Accepted : 26 February 2019

Published : 26 March 2019

DOI : https://doi.org/10.1186/s41073-019-0064-8

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Periodicals as topic
• Narrative review articles
• Non-systematic review articles
• Reliability
• Item-total correlation
• Internal consistency
• Cronbach’s alpha
• Intra-class correlation coefficient

Research Integrity and Peer Review

ISSN: 2058-8615

• Submission enquiries: [email protected]
• General enquiries: [email protected]

Literature review methods

Critical appraisal, assess for quality.

Critical appraisal refers to the process of judging the validity and quality of a research paper. Because your review will be a synthesis of the research conducted by others, it is important to consider major points about the studies you include such as: 

• Is the study valid?
• What are the results, and are they significant or trustworthy?
• Are the results applicable to your research question?

Critical appraisal tools

Several checklists or tools are available to help you assess the quality of studies to be included in your review.

• AACODS  checklist for appraising grey literature
• CASP  includes checklists for systematic reviews, RCTs, qualitative studies etc.
• Centre for Evidence Based Medicine  appraisal worksheets to appraise the reliability, importance and applicability of clinical evidence
• Joanna Briggs Institute critical appraisal tools  to assess the trustworthiness, relevance and results of published papers
• RoBiS  tool for assessing the risk of bias in systematic reviews (rather than in primary studies)
• RoB 2  Cochrane risk-of-bias tool for randomized trials

Additional resources

•    Greenhalgh, T. (2019). How to read a paper: the basics of evidence-based medicine and healthcare (6th ed). Wiley.  
• << Previous: Screening
• Next: Synthesis analysis >>

Contact the library

• Ask the library
• Book time for search help
• Suggest an acquisition
• 036-10 10 10
• Follow us on Instagram
• Follow us on Facebook

Visiting address

University library Jönköping University campus, building C Gjuterigatan 5 553 18 Jönköping

• Delivery addresses

Opening hours

• Mondays 8 – 20
• Tuesdays 8 – 20
• Wednesdays 8 – 20
• Thursdays 8 – 20
• Fridays 8 – 18
• Saturdays 11 – 15
• Sundays Closed

See more opening hours .

Log in using your username and password

• Search More Search for this keyword Advanced search
• Latest content
• Current issue
• For authors
• New editors
• BMJ Journals More You are viewing from: Google Indexer

You are here

• Online First
• Effectiveness of exercise-based sports injury prevention programmes in reducing injury rates in adolescents and their implementation in the community: a mixed-methods systematic review
• Article Text
• Article info
• Citation Tools
• Rapid Responses
• Article metrics

• http://orcid.org/0009-0009-2026-8745 Zhe Xin Zhang 1 ,
• http://orcid.org/0009-0009-6336-7137 Joseph Lai 2 ,
• http://orcid.org/0000-0001-7976-668X Liang Shen 3 ,
• http://orcid.org/0000-0001-7083-7242 Lingaraj Krishna 4 , 5
• 1 Alice Lee Centre for Nursing Studies , National University of Singapore Yong Loo Lin School of Medicine , Singapore
• 2 National University of Singapore Yong Loo Lin School of Medicine , Singapore
• 3 Biostatistics Unit , National University of Singapore Yong Loo Lin School of Medicine , Singapore
• 4 Orthopaedic and Hand Surgery Partners Pte Ltd , Singapore
• 5 Division of Sports, Shoulder & Elbow Surgery, Department of Orthopaedic Surgery , National University Hospital , Singapore
• Correspondence to Zhe Xin Zhang, Alice Lee Centre for Nursing Studies, National University of Singapore Yong Loo Lin School of Medicine, Singapore, Singapore; zhexinnn{at}gmail.com

Objective Despite evidence supporting the efficacy of sport injury prevention programmes (SIPPs) in adolescents, implementation of SIPPs in community settings is low. This review aims to synthesise and integrate evidence on the efficacy of exercise-based SIPPs in reducing injury rates in adolescents with implementation strategies for such programmes in the community.

Design A systematic review with meta-analysis, narrative synthesis and meta-aggregation was conducted, followed by a convergent segregated approach to integrate the findings. Sensitivity and subgroup analyses were conducted. Study appraisal was performed using Joanna Briggs Institute Critical Appraisal Checklists and Mixed Methods Appraisal Tool.

Data sources Literature search of nine databases was carried out to identify studies in English from January 2012 to December 2022.

Eligibility criteria Included were randomised controlled trials (RCTs), qualitative or mixed-methods studies. Population included adolescents (10–19 years). Interventions included SIPPs. Outcomes were injury rate and rate ratio (IRR). Phenomena of interest were facilitators and barriers to the implementation of SIPPs.

Results 23 studies were included for analysis. Meta-analysis for 16 RCTs showed a protective effect of SIPP (IRR 0.63, 95% CI 0.53 to 0.74, p<0.00001) in adolescents. Meta-aggregation of seven qualitative/mixed-method studies revealed four sets of synthesised findings that impact implementation namely players’ perceptions and beliefs, coaches as key facilitators, organisational support and characteristics of the SIPP.

Conclusion Implementation of SIPPs provides a 37% risk reduction in adolescents but requires targeting key stakeholders through a top-down multifaceted approach for its efficacy to be translated. Future research should investigate the effectiveness of SIPPs and implementation strategies in adolescents in community settings.

• Sporting injuries
• Preventive Medicine
• Athletic Injuries

Data availability statement

All data relevant to the study are included in the article or uploaded as online supplemental information.

https://doi.org/10.1136/bjsports-2023-107717

Statistics from Altmetric.com

Request permissions.

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

WHAT IS ALREADY KNOWN

Sports injury prevention programmes (SIPPs) are efficacious in reducing injury rates in children and adolescents, but the adoption and sustainability of SIPPs in the community remain low.

SIPP implementation in adolescents is complex, and research is needed to better understand the implementation context and promote evidence translation.

WHAT ARE THE NEW FINDINGS

This is the first review to synthesise qualitative evidence specifically regarding implementation of SIPPs in adolescents. Corroborating findings with results from quantitative analysis showing a 37% overall sports injury risk reduction reveals that key stakeholders at multiple ecological levels (organisations, coaches and players) need to be engaged to drive implementation in the community. The adaptability and user-friendliness of SIPPs also play a key role.

Due to the hierarchy of responsibility among stakeholders involved in adolescent SIPP implementation, a top-down approach to implementation would be most optimal, beginning at organisations that are best positioned to support and drive change and have downstream effects on other stakeholders.

Future studies should use findings from this paper to develop and apply implementation strategies targeted at key stakeholders and evaluate the effectiveness of SIPPs in community settings using the Reach Effectiveness Adoption Implementation Maintenance Sports Setting Matrix (RE-AIM SSM).

Introduction

Rise in popularity of sports from younger age.

Active participation in sports from young both recreationally and competitively is increasing worldwide, becoming an integral part of the life of many adolescents. 1 2 This is shown to also translate to a higher level of physical activity (PA) when older, promoting a lifelong active lifestyle. 3 Living a physically active lifestyle from young brings important immediate and long-term health benefits such as better cardiorespiratory and mental health while improving neuromuscular fitness and decreasing future risk of chronic diseases. 4–6

However, this is associated with an increased risk of sports injuries where the risk is highest during adolescence (ages 10–19). 7 8 Sports and PA are also the leading causes of injury in adolescents, accounting for >30% of all adolescents’ injuries compared with just 9% of injuries in adults. 6 9 The increased intensity and frequency of sports training and competition from young in recent years contribute to this higher sports injury rate in adolescents. 10

Sports injuries are associated with an increased risk of physical and psychosocial health issues. 7 11 Current and future participation in sports and PA decreases as a result, which leads to the loss of health benefits that come with an active lifestyle. 7 12 There are also substantial economic and individual consequences due to high costs of treatment and opportunity costs from possible extensive periods of immobility. 13 On a societal level, an increased burden is placed on public healthcare due to the higher volume of hospitalisation and the various resources required for injury treatment. 14 Hence, finding ways to reduce the sports injury rate in adolescents is of utmost priority. 4

Importance of starting sports injury prevention from adolescence

Emery et al 6 suggest that while it is not possible to entirely prevent sports injuries in adolescents, effective exercise-based Sports Injury Prevention Programmes (SIPPs) can reduce the rate and severity of sports injuries. This has many downstream benefits like reducing the various costs mentioned. 11 15 Injury prevention efforts are also recommended by Myer et al 16 to be started from adolescence to maximise efficacy as it was discovered in their meta-analysis that the age of the athlete affects the efficacy of SIPPs in reducing anterior cruciate ligament (ACL) injuries.

Rationale for this mixed-methods review

While there are four similar reviews on the efficacy of SIPP in reducing injury rates involving adolescents, there are some key differences from this review in terms of the target population. 4 6 11 17 This review looks at adolescents only in all sports but the reviews by Ding et al 17 and Rössler et al 4 looked at not only adolescents but children too, while Soomro et al 11 and Emery et al 6 limited studies to team sports. Ding et al 17 also restricted SIPP to warm-up exercises only.

Despite the multitude of scientific evidence collated in the systematic reviews supporting the efficacy of SIPPs in adolescents, implementation in real-world settings and actual public health impact remains limited. 18–20 The injury rate among adolescent athletes continues to rise and many studies have shown that SIPPs are still not part of training routines and sports practices across various populations. 7 21 22

This lack of adoption of SIPPs for adolescents highlights a necessary change in research focus and methods. 20 Understanding the factors that support and inhibit the long-term adoption of SIPPs in target populations from different perspectives by examining qualitative literature is important in influencing behavioural change in adolescents and stakeholders and reducing the research-to-practice gap. 18 19 23 Due to the complexity of sports injury prevention, multiple research questions and an integration of qualitative and quantitative evidence is optimal. 24 A mixed-methods approach broadens the scope of this review compared to previous systematic reviews by Emery et al 6 and Soomro et al 11 , allowing for a more in-depth exploration of injury prevention in adolescents. 25 26 It provides the opportunity to corroborate findings of SIPP efficacy with insights into the barriers and facilitators of SIPP implementation and maintenance, providing more meaningful evidence to inform SIPPs practice and policy in adolescents. 27 To the best of our knowledge, no such review has been conducted before on this topic.

Review objective

The objective of this mixed-methods systematic review is to update and synthesise evidence on the efficacy of exercise-based SIPPs in adolescents while integrating it with research on implementation context for injury prevention to promote evidence translation and improve effectiveness. This review seeks to answer the following review questions:

What is the efficacy of the various exercise-based SIPPs in reducing injury rates in adolescents in the community setting?

What are the facilitators (enablers) and barriers to the implementation of SIPPs in adolescents in the community setting?

This systematic review was conducted in accordance with Joanna Briggs Institute (JBI) methodology for mixed-methods systematic reviews and reported following the updated Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) 2020 checklist. 28 29 The study protocol is registered on PROSPERO (CRD 42023403096).

Information sources and search strategy

A search of six databases (PubMed, EMBASE, Cochrane Library, CINAHL (EBSCOhost), Scopus and Web of Science) was performed to identify relevant studies from January 2012 to December 2022. Google Scholar, ProQuest Thesis and Dissertations and ISRTCN were searched for grey literature. The reference list of all selected studies was also screened for additional studies not previously identified. Keywords and index terms were broken down into three groups: population, injury type and intervention, and used to develop full search strategies for each database which can be found in online supplemental file 2:appendix A .

Supplemental material

Selection of studies.

Inclusion criteria for quantitative component:

Participants were between 10 and 19 years old and participated in sports. 8

SIPPs are exercise based and implemented across a minimum period of 3 months. 30

Investigated outcomes related to sports injury (injury rate/ratio (IRR) or total number of injuries).

Controls were either usual training/warmups, alternative forms of injury prevention or no intervention.

Only randomised controlled trials (RCTs) as it is the most ideal to examine the cause-and-effect relationship between interventions and measured outcomes. 31

Studies were excluded if adolescent age-group outcomes were unable to be separately extracted or incomplete.

Inclusion criteria for qualitative component:

Studies that investigated barriers and facilitators in sports injury prevention in adolescents.

Is in the community setting.

Qualitative studies with study designs such as phenomenology, grounded theory, ethnography and qualitative descriptive. Mixed-methods studies where qualitative component can be clearly extracted.

Only English language studies were included due to the lack of reviewer proficiency in other languages and resource limitations. Since there were previous meta-analyses on the efficacy of SIPP in adolescents, 6 11 and it is only in the past decade that research emphasis has been placed on understanding the implementation of sports injury prevention, 32–34 only studies published from 2012 to 2022 were included to retrieve the most current evidence of SIPPs efficacy and insights into its implementation for adolescents.

Data extraction and quality assessment

Quantitative data extracted included specific details about the populations, intervention characteristics and outcomes of relevance to review objective ( online supplemental file 2:appendix C ). Extracted qualitative data (findings) with their corresponding illustrations were assigned a level of credibility (‘unequivocal’, ‘credible’ and ‘not supported’) and grouped into themes. Authors of papers were contacted for additional or missing data, where required.

Assessment of selected studies was done independently by two reviewers (ZXZ and JL) using the standardised JBI Critical Appraisal Checklists for RCTs and qualitative studies. 35 36 The Mixed Methods Appraisal Tool (MMAT) tool was used for mixed-methods studies instead as JBI does not have a critical appraisal tool for it. 37 Disagreements regarding appraisal were resolved through discussion between both reviewers and with a third reviewer (LK) when consensus could not be reached. All studies, regardless of methodological quality, underwent data extraction and synthesis to consolidate all available evidence to enhance the rigour of the synthesis and provide further insights into sports injury prevention. 38 39

Data analyses and synthesis

A mixed-methods convergent segregated approach was used for synthesis and integration. 28 Quantitative and qualitative synthesis was done separately followed by integration of the resultant quantitative and qualitative evidence ( online supplemental file 2:appendix D ).

Quantitative synthesis

Study data were pooled with statistical meta-analysis using RevMan V.5.4. IRR with 95% CIs was used to measure the effect size of each study. Natural logarithmic transformation of all IRR was conducted and the generic inverse variance random-effects model was used for the statistical analysis. 40 Narrative synthesis was used to present outcomes when meta-analysis was not possible.

Subgroup analyses based on session duration, implementation strategies, intervention type, intervention focus and player compliance were conducted to help examine sources of heterogeneity and identify potentially influential moderating factors on the efficacy of SIPP. 41 Sensitivity analysis was also conducted to examine the source of heterogeneity. A funnel plot was generated using RevMan to assess publication bias.

Qualitative synthesis

Qualitative findings were pooled using the meta-aggregation approach that involves an iterative approach of categorising findings based on similarity in meaning. 25 These categories were subsequently synthesised to produce a comprehensive set of synthesised findings to be used as the basis for evidence integration. 42

Integration of quantitative evidence and qualitative evidence

Through discussion by two reviewers, the quantitative and qualitative evidence was juxtaposed and organised into a line of argument to produce an overall configured analysis. 28

Refer to online supplemental file 1:Methods for more detailed methods section.

Study selection

The screening process is depicted in figure 1 using the PRISMA flow diagram. 29 An initial search yielded 9681 articles after duplicate records were removed. 9288 records were excluded based on title and 186 records based on abstracts when screened against the eligibility criteria. Eventually, the full text of 200 articles was retrieved and assessed for inclusion. Finally, 23 articles were included in this review after 177 articles were excluded for various reasons as outlined in figure 1 .

• Download figure
• Open in new tab
• Download powerpoint

PRISMA flow diagram.

Study characteristics

Of the 23 included studies, 16 were RCTs, 43–58 5 were of qualitative methodology 59–63 and 2 were mixed-methods studies. 64 65 A table summary of all the studies and their characteristics can be found in online supplemental file 2:appendix D . Among the 16 RCTs (1 is a 3-arm RCT), 9 of them looked at preventing all injuries, 2 focused on upper extremity and 6 on lower extremity injuries. In terms of SIPP content type, 12 of the studies featured comprehensive (multifaceted) SIPPs that contain a mixture of balance, plyometric, strength and neuromuscular control exercises while the other 5 were single-component SIPPs like stretching, strengthening or proprioception exercises only. Only three studies had a subanalysis on the effect of SIPP compliance on injury rates. Out of the seven qualitative and mixed-methods studies, six used either an implementation science framework or behavioural change model.

Methodological quality

Summary tables of the critical appraisal using JBI checklists and MMAT for the respective studies are presented in online supplemental file 2:appendix E . For the RCTs, blinding for researchers and participants (Q4 and 5) was either unclear or not done for almost all the studies (87.5%). However, failure to blind is unlikely to affect the objective outcomes for SIPP studies as it is known that true researcher and participant blinding is incredibly difficult in injury prevention research. 66 67 Blinding of the outcome assessor was done in 8 of the studies while true randomisation was used in 11 of the studies and unclear in the other 5. Allocation concealment was unclear in five of the studies (31.3%).

The methodological quality of all the qualitative and mixed-methods studies is generally very high (all studies >80%). However, most of the qualitative studies did not elaborate on the cultural and theoretical position of the author (80% no or unclear) and authors’ influences on the research (60% no or unclear).

Quantitative evidence: meta-analysis

Forest plot showing the effect of SIPP on IRR of adolescents.

Subgroup analyses

Subgroup analyses showed no statistically significant subgroup differences for player compliance (p=0.52), SIPP session duration (p=0.69), implementation strategies (p=0.38) and SIPP type (p=0.56), likely constrained by the small number of studies and participants in some of the subgroups 68 ( online supplemental file 2:appendix F ).

Forest plot showing subgroup analysis on intervention focus of SIPP.

Sensitivity analysis

Publication bias.

Visual inspection of the funnel plot plotted ( online supplemental file 2:appendix G ) suggested that there is a possibility of publication bias as there is a slight asymmetry. The larger-size studies were clustered symmetrically at the top around the mean effect size line, but it appears that smaller-size studies that show no protective effect of SIPP are missing. 69

Narrative summary

The secondary outcome of investigating the effect of player compliance to SIPP on injury rate was summarised using narrative synthesis as statistical pooling was not possible. Three studies examined this and all concluded that compliance is key to the effectiveness of SIPP in reducing injury rate. 50 55 57 Two of the studies further found that player compliance to SIPP deteriorated significantly over the season and there is a need for research on the effective maintenance and implementation of SIPP. 50 57

Qualitative evidence: meta-aggregation

Four sets of findings were synthesised from meta-aggregation and the process is illustrated in online supplemental file 2:appendix K .

Synthesised Finding 1: players’ perceptions and beliefs influence their motivation to adopt SIPPs

Players are the end-users of SIPP so understanding their perspectives and showing them the importance of SIPP is key to getting them to ‘buy-in’ and be on board with the programme. 59 62 65 Players must have an idea of their susceptibility to injury before they will see the need for and the benefit of such SIPP. 62 64 They often find SIPP boring and irrelevant to their sports and training. 61 65 It often takes personal experience with injury for them to understand and be motivated to adopt it. 60 65 (refer to online supplemental file 2:appendix H )

Synthesised Finding 2: coaches/teachers are the key SIPP deliverers

Coaches and teachers are the key deliverers of SIPP ( table 1 ) as they are the on-the-ground facilitators who spend the most time with the players and have direct influence over them. 62 64 65 Coaches also have full control over the structure of the training programme, so how they carry out the SIPP in training and how much emphasis, effort they put in and time they allocate makes a big difference in the eventual effect of the SIPP. 59 60 For coaches to prioritise SIPP and implement it well, they must be aware of the existence of such programmes, believe in the importance and effectiveness of SIPP for their players and have sufficient self-efficacy to implement SIPP properly. 61–63 Coaches’ motivation directly influences player motivation. 62

• View inline

Synthesised Finding 2

Synthesised Finding 3: organisational commitment and support are crucial to the successful implementation of SIPP

It is crucial that there is a readiness for implementation of SIPP from the top-down where the organisation (club, federation, etc) shows commitment to facilitate the implementation through various ways, such as endorsing the SIPP and making a directive to implement it. 59 64 65 Other ways include actively publicising and promoting SIPP to various stakeholders and providing training and resources for coaches to improve their knowledge and self-efficacy regarding SIPP. 61 62 65 Organisations should also consider making injury prevention education a mandatory part of coaching education to really effect a change in attitude among coaches. 59 64 (refer to online supplemental file 2:appendix I )

Synthesised Finding 4: the characteristics of the SIPP itself influence the adoption and use of the programme

The characteristics of the SIPP also affect its adoption by coaches and players. It is crucial for SIPP to be adaptable and modifiable to suit the needs of different sports and contexts. 63 65 The opportunity for coaches to be creative and integrate SIPP exercises into game drills will make it more fun and engaging. 59 61 64 For SIPP to be attractive, it has to be easy to implement and requires minimum time and effort or any additional resources, lowering the barrier to adopting or implementing it. 59 62 63 (refer to online supplemental file 2:appendix J )

Mixed-methods integration of evidence

Synthesised finding 2 (coaches are key facilitators) and 3 (organisational support is crucial) explained why the demonstrated efficacy of SIPP in reducing injuries found in the main meta-analysis is not translated to practice. 18 All the studies analysed were conducted under RCT conditions which are ideal and highly controlled, where most had experts conducting mandatory prestudy workshops and training for the coaches and players on the proper use of SIPP which is equivalent to organisational support and training to improve knowledge and self-efficacy. 70 This is shown by synthesised findings 2 and 3 to be major facilitators of SIPP implementation. Additional resources like pamphlets and video instructions were also provided to the coaches in many of the studies while some even had physiotherapists to facilitate and provide feedback for sessions. All these were factors identified by the qualitative synthesis to improve the implementation and effectiveness of SIPP but not implemented often enough in practice.

Narrative synthesis finding regarding player compliance is congruent with synthesised finding 1 (players’ perceptions and beliefs) as young adolescent players often do not see the need for SIPP due to their ‘sense of invincibility’ and soon find SIPP boring and irrelevant, explaining the decrease in compliance over the season.

While the subgroup analyses for duration of SIPP and implementation strategies were not statistically significant, there was indication that shorter SIPPs (≤15 min IRR=0.61 vs >15 min IRR=0.66) and those with implementation strategies (workshop+supervision IRR=0.54 vs workshop IRR=0.67 vs no strategy IRR=0.72) were potentially more efficacious ( online supplemental file 2:appendix F ) and is supported by synthesised findings 3 and 4.

This review synthesised findings from 23 studies to examine the efficacy of SIPPs in reducing injury rates in adolescents and understanding the translation of this efficacy to real-world settings. The Translating Research into Injury Prevention Practice (TRIPP) framework is used to provide a comprehensive understanding of the current adolescent sports injury prevention literature and how this mixed-methods systematic review contributes and advances it 71 ( table 2 ). Since the most recent meta-analysis in 2016 by Soomro et al , 11 there have been 13 newly published RCTs. Therefore, the quantitative aspect of our research was performed on the most recent decade of RCTs to include all these RCTs and update TRIPP Stage 4 with the most current evidence on the protective efficacy of all types of exercise-based SIPP in adolescents in all sports. The findings of our research revealed a significant overall protective effect of exercise-based SIPP in adolescents, where the injury rate was reduced by 37%. This is similar to results from previous meta-analyses both on adolescent team sports by Soomro et al 11 (exercise-based SIPPs) and Emery et al 6 (neuromuscular SIPPs), which found an estimated 40% and 36% reduction in all injuries and lower extremity injuries, respectively.

Adapted from Translating Research into Injury Prevention Practice (TRIPP) Framework of Caroline Finch 71 and Emery et al ’s Ecological model 77

When compared with findings from previous related systematic reviews that looked at either different age groups or types of SIPPs, the efficacy of SIPPs demonstrated in reducing injury rates is consistent. For example, the meta-analysis by Ding et al 17 looked at the effectiveness of warm-up-only interventions in reducing sports injuries in adolescents and children in 2021 and found an estimated 36% reduction in injury rate. Likewise, Rössler et al , 4 which combined data from RCT and non-randomised study designs, concluded that exercise-based SIPPs can reduce injuries by around 46%. Therefore, our research reaffirms TRIPP stage 4 evidence on the benefits of long-term application of SIPP on reducing injury rates in adolescents, matching findings from systematic reviews examining that in the general population. 67 72 73

Strong evidence base of TRIPP stage 4 alone is insufficient to change the sports injury landscape in adolescents as RCTs are carried out in highly controlled environments that are not reflective of actual implementation contexts. 70 74 75 For adoption and compliance in the community to improve, there is a need to understand the implementation issues that exist in the specific context, and this is best done through qualitative research on SIPP implementation in adolescents (TRIPP stage 5). 6 32 However, there has not been any reviews that synthesised and consolidated such research and thus, the qualitative aspect of our review fills this gap in the literature.

Using ecological model and TRIPP framework to interpret mixed-methods evidence on SIPPs and improve real-life impact

Akin to numerous implementation science research done on sports injury prevention in the general population, 32 75 76 it is evident from the integrated findings that engagement of various stakeholders at multiple ecological levels is key to improving adoption and maintenance of SIPPs in adolescents. However, for this population, interpretation of the findings should be additionally guided by the ecological model proposed by Emery et al 77 as it incorporates the perspectives of multiple stakeholders and assigns a hierarchy of responsibility for the stakeholders in the implementation of SIPPs ( table 2 ).

Top-down approach is most optimal for SIPP implementation in adolescents

Despite the multitude of evidence supporting the implementation of SIPP in adolescents, there is still a lack of awareness of these SIPPs among coaches and players worldwide, 6 which is the first step of the awareness-to-adherence model by Pathman et al 78 for behaviour change. 75 In this ecological model, adolescents bear the lowest responsibility even though they are the beneficial end-users as they are deemed not cognitively developed enough to take full responsibility for their own safety in sports. 1 In contrast, organisations like national sports governing bodies and clubs at the top of the hierarchy who have the power to effect the most change will bear the most responsibility. Therefore, it is optimal to start from the top (synthesised finding 3) as successful implementation of SIPPs especially in adolescents relies heavily on organisational commitment and structures and resources being put in place to support the delivery of SIPPs.

Organisations are found in several studies included to be best positioned to drive changes to this as they are the ones with the reach to disseminate information and the resources to develop and provide SIPP training, raising awareness of SIPP among the stakeholders. They even have the option to mandate injury prevention training as part of coaching education, along with supporting it with directives that require coaches to implement SIPP as part of training sessions. Such emphasis and commitment to implementation from governing bodies will have downstream effect on all the stakeholders from clubs to coaches (improved awareness and self-efficacy) and the players (SIPP is normalised as part of training), improving implementation. The higher than usual implementation rate of Prep-to-Play PRO in women’s Australian Football 76 and FIFA 11+ in Switzerland soccer clubs 79 where both had the governing organisation’s full commitment and support are some examples of this impact. This is congruent with general implementation research that has long highlighted organisation and leadership as core drivers of effective implementation. 80

Coaches are key delivery agents

In addition, coaches have proven to be key delivery agents of SIPP due to their unparalleled influence over adolescents and their training programmes, consistent with findings from studies on coach education used in the BokSmart Safe Six SIPP. 81 82 A scoping review by Guilfoyle et al 83 on coaches’ role in youth SIPPs supported our findings that coaches’ competency in delivering SIPP (self-efficacy) and their belief in the value of SIPPs (buy-in) are the two main factors that promote the implementation. Therefore, both the literature and our integrated findings found that organisations need to constantly support coaches with resources and trainings like workshops which will not only improve their confidence in delivering SIPP (self-efficacy) but also educate them on importance of SIPPs to create buy-in (behaviour change). 76 84 This helps facilitate sustained implementation. 83 Furthermore, coaches’ willingness to integrate SIPP into training by prioritising and allocating more training time to SIPP will often influence players’ own thoughts and habits in using SIPPs. This aids in their own long-term compliance to SIPP which tends to decrease over a season.

These findings illustrate how a top-down approach starting at organisations will maximise implementation efforts in a target population that is young and impressionable. 85

Nature of SIPP

Outside of the ecological model, the nature of the SIPP is another key factor in the implementation. Coaches in the community for this age group vary greatly in expertise and experience, and also differ in commitment level (from full-time professional to part-time voluntary). 32 86 Hence, it is important for SIPP to be user-friendly and easy to implement where not a lot of resources are required and exercises not too difficult or long. However, it is also important for SIPPs to be adaptable in nature to cater to a range of contexts and different end-users. 32 Adaptability of the exercises allows coaches to adjust and progress according to the evolving needs of the team/individual. It also makes SIPP more engaging when coaches are able to integrate it into drills and game skills which is important as adolescents do not always see the necessity for SIPP and finds it boring.

Strengths and limitations

This review has a few limitations. First, there is rather substantial heterogeneity in the included studies, possibly due to the methodological variations in the nature of participants, outcomes measured due to different definitions of sports injury and types of intervention in the studies included. Moreover, non-English studies were excluded which may have led to potential bias and a date limit of 10 years being set could have excluded some older qualitative studies.

To the best of our knowledge, this review is the first in the field that integrated TRIPP stages 4 and 5 ( table 2 ), 71 corroborating the most rigorous up-to-date scientific evidence on the protective efficacy of SIPP with insights into the specific implementation of SIPP in adolescents, promoting evidence translation. 19

Direction and implications for future research

Since this review has synthesised both TRIPP stages 4 and 5 evidence, it advances adolescent sports injury research to the next stage, which is to focus on implementation science and evaluate the effectiveness of SIPPs in uncontrolled ‘real-world’ settings with implementation strategies applied to obtain more representative results of the impact of SIPPs in reducing injury rates in adolescents. 70 75 This is done through TRIPP stage 6 effectiveness studies that the current literature lacks. 71 Thus, our findings help researchers to develop context-specific implementation strategies in partnership with relevant stakeholders and be used in such studies. 75 TRIPP stage 6 studies should also consider using the Reach Effectiveness Adoption Implementation Maintenance Sports Setting Matrix (RE-AIM SSM) ( table 3 ) as it provides a framework 70 to navigate the complex multilevel nature of SIPP implementation, 75 guide the planning of appropriate strategies and thoroughly evaluate the impact of SIPPs across the entire hierarchy of stakeholders. 70 77 The mixed-methods study by Bruder et al 76 on SIPP for women’s elite Australian Football provides a great example of this.

RE-AIM Sports Setting Matrix (RE-AIM SSM) adapted from Finch and Donaldson 70

Lastly, there should be a development of a consensus statement on the reporting standards for SIPPs where injury definition, exposure monitoring, assessment of outcome etc. are standardised to ensure homogeneity in study design. This would help in identifying moderating factors and a clearer interpretation of results.

The efficacy of SIPP in reducing injury rates in adolescents is reaffirmed by the past decade of research but translation of this efficacy to ‘real-world’ effectiveness is impaired by poor adoption and compliance in the community. 74 Synthesising qualitative findings on the implementation of SIPP in adolescents and integrating it with the quantitative results through the convergent segregated approach provided many valuable insights, indicating that key stakeholders at multiple ecological levels (organisations, coaches, players) need to be engaged to drive implementation. Due to the hierarchy of responsibility for stakeholders involved in adolescent SIPP implementation, employing a top-down approach by targeting organisations (governing bodies) first as they are best positioned to support and drive change, followed by coaches (delivery agents) through increasing competence and buy-in and then players (end-users) would be the most optimal and allows for downstream effects. 23 77 85 Meanwhile, the adaptability and user-friendliness of SIPPs are also crucial in improving implementation. Future effectiveness studies evaluating SIPPs in ‘real-world’ contexts is the last part of the research process in achieving the ultimate goal of improving adoption and maintenance of efficacious SIPP in respective sporting communities to yield the full benefits of SIPPs and thus sports. 71 75

Ethics statements

Patient consent for publication.

Not applicable.

Acknowledgments

We thank Dr Siriwan Lim and A/Prof Piyanee Yobas from National University of Singapore (NUS) for their guidance in the conception of this research as thesis advisors and Chan Pao Yi (NUS) for her comments on the manuscript.

• Maffulli N ,
• Hagel B , et al
• Kjønniksen L ,
• Anderssen N ,
• Rössler R ,
• Verhagen E , et al
• Dobbins M ,
• DeCorby K , et al
• Whittaker JL , et al
• Benjaminse A ,
• Robards F ,
• McKenna J , et al
• Sanders R ,
• Hackett D , et al
• Gougoulias N , et al
• Verhagen E ,
• Rommers N , et al
• Sugimoto D ,
• Thomas S , et al
• Smith DM , et al
• Minnig MC ,
• Hawkinson L ,
• Root HJ , et al
• Lininger MR ,
• DiStefano LJ
• Norcross MF ,
• Johnson ST ,
• Bovbjerg VE , et al
• Bruinsma A , et al
• Methods LDM
• Lizarondo L ,
• Carrier J , et al
• Bressan V ,
• Bagnasco A ,
• Aleo G , et al
• Pearson A ,
• Bath-Hextall F , et al
• McKenzie JE ,
• Bossuyt PM , et al
• Clarke GM ,
• Wolters AT , et al
• Hariton E ,
• Locascio JJ
• McLaren SJ ,
• Bolling C ,
• van Mechelen W ,
• Pasman HR , et al
• Donaldson A ,
• Lockwood C ,
• Porritt K ,
• Munn Z , et al
• Tufanaru C ,
• Aromataris E , et al
• Fàbregues S ,
• Bartlett G , et al
• McDonagh M ,
• Peterson K ,
• Raina P , et al
• Stephenson M , et al
• Higgins JP ,
• Altman DG , et al
• Achenbach L ,
• Krutsch V ,
• Weber J , et al
• Åkerlund I ,
• Sonesson S , et al
• Hägglund M ,
• Waldén M , et al
• van den Berg C ,
• Richmond SA , et al
• Farhan AF ,
• J. Stephany M ,
• K. Mahammed S
• Akasaka K ,
• Otsudo T , et al
• Leppänen M ,
• Vasankari T , et al
• Hislop MD ,
• Stokes KA ,
• Williams S , et al
• Loppini M ,
• Berton A , et al
• Owoeye OBA ,
• Akinbo SRA ,
• Tella BA , et al
• Richmond SA ,
• Doyle-Baker PK , et al
• Nakamura E ,
• Suzuki T , et al
• Mizoguchi Y ,
• Kimura F , et al
• Atroshi I ,
• Magnusson H , et al
• Zouita ABM ,
• Kebsi W , et al
• Watkins R ,
• Stokes KA , et al
• Standage M ,
• Hagger MS , et al
• Lindblom H ,
• Carlfjord S ,
• Munoz-Plaza C ,
• Davis A , et al
• Macpherson A , et al
• Ageberg E ,
• Lucander K , et al
• Myklebust G , et al
• Jacobs JC ,
• Kim J , et al
• Lauersen JB ,
• Bertelsen DM ,
• Andersen LB
• Richardson M ,
• Sterne JAC ,
• Donaldson A
• Hübscher M ,
• Pfeifer K , et al
• Aaltonen S ,
• Parkkari J , et al
• Verhagen EALM , et al
• Rauvola RS ,
• Brownson RC
• Bruder AM ,
• Patterson BE ,
• Crossley KM , et al
• Morrongiello BA
• Pathman DE ,
• Konrad TR ,
• Freed GL , et al
• Lamprecht M ,
• Stamm H , et al
• Bertram R ,
• Gardner-Lubbe S ,
• Lambert MI , et al
• Lambert M , et al
• Guilfoyle L ,
• O’Sullivan K , et al
• McGlashan AJ ,
• Didymus FF ,

Supplementary materials

Supplementary data.

This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.

• Data supplement 1
• Data supplement 2

Contributors ZXZ conceived the systematic review and was responsible for the planning, data search, data extraction, appraisal of studies, data analysis and writing of manuscript and is the guarantor. JL contributed to the data search, data extraction, appraisal of studies and writing of manuscript. LS contributed to the data analysis and interpretation of the data. LK participated in the appraisal of studies, interpretation of the data and writing of manuscript.

Funding The authors have not declared a specific grant for this research from any funding agency in the public, commercial or not-for-profit sectors.

Competing interests None declared.

Provenance and peer review Not commissioned; externally peer reviewed.

Supplemental material This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.

Read the full text or download the PDF:

Identifying the practice patterns of optometrists in providing falls prevention management: A mixed-methods systematic review protocol

• Find this author on Google Scholar
• Find this author on PubMed
• Search for this author on this site
• ORCID record for Si Ye Lee
• For correspondence: [email protected]
• ORCID record for Khyber Alam
• ORCID record for Jason Charng
• ORCID record for Hamed Niyazmand
• ORCID record for Jingyi Chen
• ORCID record for Anne-Marie Hill
• Info/History
• Preview PDF

Objective The objective of this systematic review is to synthesise the best available evidence for optometrists practice patterns in providing falls prevention management. Introduction Falls remain the main cause of injury related hospitalisation and mortality in Australia and worldwide, significantly affecting older adults. The increased risk of comorbidities, including visual impairment in this cohort is linked to a higher incidence of falls. Despite being primary eye care practitioners, community optometrists may not consistently integrate falls prevention strategies into their practice. Furthermore, the extent to which they adhere to evidence-based recommendations for falls management remains unclear. Inclusion criteria The review will include optometrists, in regions where optometry is a regulated profession, and report their understanding and practice patterns in delivering falls prevention management to older community dwelling adults. Qualitative, quantitative, and mixed methods studies will be eligible for inclusion. It is envisioned that most studies will be qualitative. Studies published in English and those published from 1980 onwards will be eligible for inclusion since published evidence for falls prevention began to increase sharply around this time. Methods The review will follow the JBI guidelines for mixed methods systematic reviews and will be developed and reported in accordance with PRISMA P guidelines. Databases that will be searched are Excerpta Medica Database (Embase), Scopus, Cumulative Index to Nursing and Allied Health Literature (CINAHL) Complete, and OVID MEDLINE. Grey literature will be searched through Conference Proceedings Citation Index (Web of Science), Google Scholar, and ProQuest Dissertations databases. Two reviewers will independently conduct all screening and critical appraisal. The reviewers will screen all articles titles and abstracts retrieved from the searches to determine potential eligibility. All full-text articles considered relevant will then be assessed for final eligibility for inclusion. The final included articles will be assessed for methodological rigour using the JBI SUMARI critical appraisal tools, subsequently, all relevant data will be extracted. Discrepancies at any stage of the procedures will be resolved through discussion and consensus with a third senior researcher. A convergent integrated approach to synthesising and integrating the quantitative and qualitative data will be followed. Review registration CRD42024539668 Keywords Accidental falls, Optometry, Practice Patterns, Aged

Competing Interest Statement

The authors have declared no competing interest.

Funding Statement

Anne-Marie Hill is supported by a National Health and Medical Research Council (NHMRC) of Australia Investigator (EL2) award (GNT1174179) and the Royal Perth Hospital Research Foundation. Si Ye Lee is conducting this research with the support of an Australian Government Research Training Program Fees Offset scholarship and is a recipient of a Perth Eye Foundation scholarship through the University of Western Australia.

Author Declarations

I confirm all relevant ethical guidelines have been followed, and any necessary IRB and/or ethics committee approvals have been obtained.

I confirm that all necessary patient/participant consent has been obtained and the appropriate institutional forms have been archived, and that any patient/participant/sample identifiers included were not known to anyone (e.g., hospital staff, patients or participants themselves) outside the research group so cannot be used to identify individuals.

I understand that all clinical trials and any other prospective interventional studies must be registered with an ICMJE-approved registry, such as ClinicalTrials.gov. I confirm that any such study reported in the manuscript has been registered and the trial registration ID is provided (note: if posting a prospective study registered retrospectively, please provide a statement in the trial ID field explaining why the study was not registered in advance).

I have followed all appropriate research reporting guidelines, such as any relevant EQUATOR Network research reporting checklist(s) and other pertinent material, if applicable.

Data Availability

All data produced in the present work are contained in the manuscript.

View the discussion thread.

Thank you for your interest in spreading the word about medRxiv.

NOTE: Your email address is requested solely to identify you as the sender of this article.

Citation Manager Formats

• EndNote (tagged)
• EndNote 8 (xml)
• RefWorks Tagged
• Ref Manager
• Tweet Widget
• Facebook Like
• Google Plus One
• Addiction Medicine (324)
• Allergy and Immunology (628)
• Anesthesia (165)
• Cardiovascular Medicine (2378)
• Dentistry and Oral Medicine (289)
• Dermatology (207)
• Emergency Medicine (379)
• Endocrinology (including Diabetes Mellitus and Metabolic Disease) (837)
• Epidemiology (11775)
• Forensic Medicine (10)
• Gastroenterology (703)
• Genetic and Genomic Medicine (3746)
• Geriatric Medicine (350)
• Health Economics (634)
• Health Informatics (2399)
• Health Policy (933)
• Health Systems and Quality Improvement (898)
• Hematology (341)
• HIV/AIDS (782)
• Infectious Diseases (except HIV/AIDS) (13318)
• Intensive Care and Critical Care Medicine (768)
• Medical Education (365)
• Medical Ethics (105)
• Nephrology (398)
• Neurology (3508)
• Nursing (198)
• Nutrition (526)
• Obstetrics and Gynecology (674)
• Occupational and Environmental Health (664)
• Oncology (1824)
• Ophthalmology (538)
• Orthopedics (219)
• Otolaryngology (287)
• Pain Medicine (233)
• Palliative Medicine (66)
• Pathology (446)
• Pediatrics (1035)
• Pharmacology and Therapeutics (426)
• Primary Care Research (420)
• Psychiatry and Clinical Psychology (3178)
• Public and Global Health (6145)
• Radiology and Imaging (1280)
• Rehabilitation Medicine and Physical Therapy (747)
• Respiratory Medicine (828)
• Rheumatology (379)
• Sexual and Reproductive Health (372)
• Sports Medicine (323)
• Surgery (402)
• Toxicology (50)
• Transplantation (172)
• Urology (146)