research papers on analytical hierarchy process

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

• Advanced Search
• Journal List
• BMC Med Inform Decis Mak

Applying the Analytic Hierarchy Process in healthcare research: A systematic literature review and evaluation of reporting

Katharina schmidt.

Center for Health Economics Research Hannover (CHERH), Leibniz University of Hanover, Otto-Brenner-Str. 1, 30159 Hannover, Germany

Ines Aumann

Ines hollander.

Institute for Risk and Insurance, Leibniz University of Hanover, Otto-Brenner-Str. 1, 30159 Hannover, Germany

Kathrin Damm

J.-matthias graf von der schulenburg.

The Analytic Hierarchy Process (AHP), developed by Saaty in the late 1970s, is one of the methods for multi-criteria decision making. The AHP disaggregates a complex decision problem into different hierarchical levels. The weight for each criterion and alternative are judged in pairwise comparisons and priorities are calculated by the Eigenvector method. The slowly increasing application of the AHP was the motivation for this study to explore the current state of its methodology in the healthcare context.

A systematic literature review was conducted by searching the Pubmed and Web of Science databases for articles with the following keywords in their titles or abstracts: “Analytic Hierarchy Process,” “Analytical Hierarchy Process,” “multi-criteria decision analysis,” “multiple criteria decision,” “stated preference,” and “pairwise comparison.” In addition, we developed reporting criteria to indicate whether the authors reported important aspects and evaluated the resulting studies’ reporting.

The systematic review resulted in 121 articles. The number of studies applying AHP has increased since 2005. Most studies were from Asia (almost 30 %), followed by the US (25.6 %). On average, the studies used 19.64 criteria throughout their hierarchical levels. Furthermore, we restricted a detailed analysis to those articles published within the last 5 years ( n  = 69). The mean of participants in these studies were 109, whereas we identified major differences in how the surveys were conducted. The evaluation of reporting showed that the mean of reported elements was about 6.75 out of 10. Thus, 12 out of 69 studies reported less than half of the criteria.

The AHP has been applied inconsistently in healthcare research. A minority of studies described all the relevant aspects. Thus, the statements in this review may be biased, as they are restricted to the information available in the papers. Hence, further research is required to discover who should be interviewed and how, how inconsistent answers should be dealt with, and how the outcome and stability of the results should be presented. In addition, we need new insights to determine which target group can best handle the challenges of the AHP.

The resources in health care systems are limited. Exacerbating this issue is the problem that many developed countries face, that is, the rising proportion of older, multimorbid patients, who serve to raise the cost of health care. Furthermore, innovations in medical care, such as equipment, pharmaceuticals, and treatment methods, are also driving up costs. German politicians have adopted new laws to manage the costs of pharmaceuticals, e.g. the Act on the Reform of the Market for Medicinal Products in 2011 (in German: AMNOG [ 1 ]). In this context, patient-relevant outcomes have drawn greater attention because the added benefit for patients determines the reimbursement price. But also, other countries are interested in reliable methods to measure benefits for patients, for example, to support Health Technology Assessments by patient preferences [ 2 , 3 ]. Therefore, while it is now important to measure the benefits and to prioritize the needs of patients, it will be even more so in the future. However, several studies have found a divergence in patients’ and physicians’ preferences or priorities regarding prevention and therapy (e.g. [ 4 – 6 ]). Thus, one mean of evaluating these preferences and bringing them into accord is to take the required perspective for the situation. In order to find appropriate methods for measuring the benefits and for prioritizing them, beside the established methods, new approaches of decision making tools are transferred from other fields of research, like the marketing sector. For all of these methods it is essential to measure the trade-off between attributes in multi-criteria decision situations for each participant or the group, and as such, adequate and understandable methods are essential.

Several methods are known for multi-criteria decision making in the field of health care, including value based methods, strategy based methods, and conjoint analyses [ 7 ]. In Germany, the Institute for Quality and Efficiency in Health Care (IQWiG) suggested two methods for multi-attribute decision making: Conjoint Analysis (CA) and the Analytic Hierarchy Process (AHP) [ 8 ]. Although they concluded that both methods are applicable for decision making, they were also confronted with methodological limitations. As the advantages and disadvantages of established methods like the CA have been discussed in a number of publications (e.g. [ 9 – 11 ]), the AHP method has received less attention. Therefore, we wanted to figure out whether the AHP method could become a good alternative in multi-criteria decision making.

Relevance and objective of the study

The Analytic Hierarchy Process (AHP) was developed by Saaty in the late 1970s and originally was applied to the marketing sector [ 12 , 13 ]. Dolan et al. were the first to apply this method to health economics research in 1989 [ 14 , 15 ]; since then, it has been accepted slowly as a method in the field of multi-criteria decision making in healthcare. Liberatore and Nydick described the importance of applying the AHP as follows: “Health care and medical decision making has been an early and on-going application area for the AHP” [ 16 ]. The AHP method was applied to different contexts, for example, the development of clinical guidelines [ 17 , 18 ] or biomedical innovations and technology development [ 19 , 20 ].

The increasing application of the AHP has been the motivation for this study to explore the current state of its methodology. The method is the basis for assessing the best instrument for each decision situation and reflecting each participant’s opinion correctly. A review provides an overview of published papers in this field. In line with De Bekker-Grob et al. [ 21 ], we provide a systematic review of the AHP. Therefore, an overview is given of the year of publication, country, and number of criteria used in the AHP (Section 3). In addition, Hummel and Ijzerman [ 22 ] analyzed the thematic field in which AHP is used. They identified the different areas of application (e.g., shared decision making, clinical guidelines, and healthcare management), number of criteria and alternatives, individual or group decisions, participants, and rating method. We focus on the methodological applications in the second step. In addition, the analyzed time horizon (2010–2015) should provide an update on Hummel and Ijzerman’s study and allow us to provide details of the most recent developments in the subject area. As in Mühlbacher’s overview [ 23 ], the field of application and the sample are inspected, although our focus remains on the current state of the research (the last 5 years) and the reporting of methodological aspects in the papers. In addition, the evaluation of studies’ reporting allows deeper insights. Therefore, we develop criteria for reporting the AHP method and determine to what extent the studies fulfill the criteria. We conclude by proposing recommended situations in which the AHP can be used.

AHP – a short introduction

As a short introduction into the method of AHP, we report the most important aspects here. We refer to detailed papers to provide deeper insights into specific methodological aspects.

The AHP disaggregates a complex decision problem into different hierarchical levels (see Saaty’s axioms for the AHP [ 24 ]). The application of an AHP is structured into six steps (see also Fig.  1 ), suggested by Dolan et al. [ 25 ] and Dolan [ 7 ], as follows: 1. define the decision goal, criteria, and alternatives, 2. rate the criteria in pairwise comparisons, 3. calculate the relative priority weights for the (sub-)criteria, 4. calculate the criteria’s global priority weights and combine the alternatives’ priorities, 5. control for inconsistency, and 6. perform sensitivity analysis.

Steps of the AHP (modeled after Dolan et al. [ 25 ] and Dolan [ 7 ]])

At the first hierarchical level, the aim of the study is defined followed by the main criteria, which can be divided further at lower levels into sub-criteria. If necessary, alternatives that contain specific combinations of characteristics can be arranged at the lowest level of the hierarchy. Although the AHP was introduced for group decisions, it may also be applied to single person decisions [ 26 ]. Pairwise comparisons at each hierarchical level present the judgments and they must be evaluated according to a scale developed by Saaty, which ranges from 9 to 1 to 9. If the alternatives consisted of subjective combinations of the criteria, the alternatives would be judged also with regard to each criterion. Saaty provided a detailed description of his scale and its intensities [ 12 ].

In order to analyze the interviews, the pairwise comparisons of (sub-)criteria at each level are displayed in ordered schemes (matrixes). An example is seen in Saaty ([ 24 ], p. 164). Only half of the matrix has to be filled in, as the other half is obtained from the reciprocal weights. The Eigenvector method (EV) is the most common means of calculating the priority vector, although other methods, such as additive normalization, weighted least-squares, logarithmic least-squares, logarithmic goal programming, and fuzzy preference programming methods, yield comparable results [ 27 ]. The EV relies on the matrix’s principle eigenvalue, which results from a process of repeated squaring and normalization (for more information, see Srdjevic [ 27 ] or Saaty [ 12 ]). The resulting local weights describe the relative priorities in relation to their parent criterion. The local weights form the global weights for the criteria through multiplication with the local weights from their parent criteria [ 24 ]. Thereby, global weights for criteria show the importance of each criterion in the overall context of the hierarchy. The priorities for the alternatives of the AHP are calculated by the sum of the particular local and global weights for each alternative [ 23 ]. For detailed information and examples concerning the calculations, see Saaty [ 28 ].

The aggregation of the individual judgments or priorities is fundamental to the outcome of the study. The first option is to have the group of participants vote by finding consensus. Another alternative is to aggregate the individual judgments. Still further, the literature suggests finding the geometric mean [ 29 ] or arithmetic mean [ 30 ]. In addition, the timing of calculating the average affects the results [ 30 ], specifically, the average of participants’ judgments or the average of participants’ global weights. Yet another option is to give special weight to one participant’s decision on the basis of that participant being an expert in the field or holding an exceptional position within the group [ 30 ]. The consistency ratio (CR) measures the uniformity of a respondent’s answers to the AHP questions. Saaty [ 24 ] describes the calculation of the CR in detail. The CR can also be calculated for a group of respondents.

Although the AHP has been applied to a variety of topics within the healthcare field, the sensitivity analyses on hierarchical decision making has received little investigation [ 31 ]. It should be noted that there are two distinct types of sensitivity analysis, that of judgments and that of priorities [ 32 ]. The former has been explained and tested by Arbel [ 33 ], Moreno-Jimenez and Vargas [ 34 ], and Sugihara and Tanaka [ 35 ]. They determined the judgments’ upper and lower bounds and articulated the preferences through preference structures. Other approaches originate from Moreno-Jimenez and Vargas [ 34 ], Triantaphyllou and Sánchez [ 36 ], Sowlati et al. [ 37 ], Masuda [ 38 ], and Huang [ 39 ]. Erkut and Tarimcilar [ 40 ] provided “a collection of practical tools for a potentially powerful sensitivity analysis in the AHP”. In addition, Altuzarra et al. [ 41 ] proposed a method for determining the stability of group decisions. If the AHP includes alternatives, the sensitivity analysis could show the effect of varying weights on the alternatives’ rankings [ 23 ]. Therefore, potential rank reversal of alternatives can be simulated. Rank reversal occurs when adding or deleting an (irrelevant) alternative leads to a shift in the previous alternatives’ ranking order [ 42 ].

This chapter is divided into two parts to introduce the methods used in this paper. The first part describes the method of the systematic review, which includes the key words and a flow chart. Further, in chapter 2.2, we describe our evaluation of reporting quality for the included studies.

Systematic literature review

The basis of this review is a systematic literature research on the Pubmed and Web of Science databases (date of research: 10/27/2015). As we focused our research question on healthcare, we did not include further databases in the other scientific fields. We searched both databases for articles with the following keywords in their titles or abstracts: “Analytic Hierarchy Process,” “Analytical Hierarchy Process,” “multi-criteria decision analysis,” “multiple criteria decision,” “stated preference,” and “pairwise comparison.” We provided the search strategy in Appendix : Table 1. It was technically not possible to search Web of Science for keywords in the abstracts. We refined the search by including only articles written in German or English and those associated with healthcare. Two independent reviewers evaluated the titles and abstracts of the resulting studies. Therefore, the criterion for inclusion was that the article is the primary source and the study used the AHP method within the healthcare setting. Additionally, we conducted a manual search to find further articles not included in the aforementioned databases. Thereafter, the two reviewers screened the full texts of the remaining articles and discussed whether to include them in the review. After reaching consensus, the important information was summarized in a table (not shown). Apart from common information, like the author, title, publication year, country, and journal, we extracted additional information regarding the study’s aim, source of criteria identification, hierarchy design, form of implementation, and analytical steps in order to conduct our analysis. The results are described in Section 3 for the entire period and in detail for the last 5 years in Subsection 3.1. The first step should give a short overview of all studies that were conducted with AHP in health care. In the second step, we reported the current state of research in more detail.

Evaluation of reporting quality

The papers identified from the last 5 years resulting from the systematic review were evaluated with regard to their reporting quality. Because there was no set standard by which to judge the AHP’s methodological issues, the evaluation of the studies’ quality was quite challenging. The before mentioned studies by De Bekker-Grob et al. [ 21 ], Hummel and Ijzerman [ 22 ], and Mühlbacher et al. [ 23 ] did not report quality criteria. However, the Consolidated Standards of Reporting Trials (CONSORT) Statement for randomized controlled trials [ 43 ] and the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) Statement [ 44 ] may provide some direction by providing checklists for transparent and complete reporting. The reason why authors should report specific aspects is the traceability of the study. Some criteria from the CONSORT Statement could be transferred to AHP studies: sample size, participants (eligibility criteria), trial designs, and statistical methods. In the case of the AHP method, the latter criterion consists of the CR, the method used to calculate the weights, the statistical software, and sensitivity analyses. Another checklist item is the description of the intervention. Transferred to the AHP method, authors should provide information about the interview process. Besides, another guideline for good research practices is published by Bridges et al. [ 9 ]. They provide a detailed checklist that is specific for conducting conjoint analyses. Since it suggests quality aspects only for those kinds of studies, the checklist cannot be used directly for our evaluation. However, we summarized the recommendations from the different fields and we obtained a simplified measurement of reporting by counting the elements that were included in the studies. Therefore, we evaluated whether the authors mentioned aspects for the following elements in their papers:

• Decision goal, criteria (and if necessary alternatives)
• Number of participants
• Type of participants (patients, potential consumers, or experts)
• Decision making by group or individually
• Scale for pairwise comparisons
• Interview process (face to face, email, questionnaire, judgments based on literature)
• Calculation of weights
• Sensitivity analysis

The last criterion was valid only for studies including alternatives. Thus, for the other papers without alternatives, we could determine only whether descriptive statistics (e.g., standard deviation, SD and confidence intervals, CI) were reported for the judgments or weights. We calculated the sum of all reported aspects for each study and present the results in Appendix : Table 2 and we show charts in Subsection 3.2. Nevertheless, we could not evaluate the content of each of the abovementioned criteria but only whether the criteria were reported in the study.

The search in Pubmed yielded to 1,956 articles and the search in Web of Science yielded to 4,829 articles, as Fig.  2 shows. Furthermore, 44 additional records were found via manual search. By screening titles and abstracts, we limited the sample to 246 articles (we excluded a total of 6,485 articles based on language or irrelevance to healthcare and we found 54 duplicates). Thereafter, we examined the full articles in order to determine whether they apply AHP to the field of healthcare. An additional 125 papers were excluded because they were not original studies or they used other stated preference methods (e.g., discrete choice experiment). In total, this process yielded to 121 relevant studies; the Appendix : Table 3 provides a complete list. We provide a brief overview of these studies to show how many studies have been published in this field and in which context the authors used the AHP. In addition, the overview presents the development and the relevance to the AHP method. In order to explore the current state of the literature, we limited the body of our research to articles published within the last 5 years. This restriction reduced the number of studies to 69. The detailed analysis of these studies’ methodologies made it necessary to reduce the number of articles.

Flow Chart of the Systematic Literature Review

For a first overview, we briefly summarized the key factors of all of the relevant articles ( n  = 121), such as their publication year, country, number of attributes, and levels.

The earliest study to use the AHP was published in 1981, but the AHP has become increasingly popular since 2005 (see also Fig.  3 ). The 2 years with the greatest number of studies published on the subject were 2011 and 2012 with nine each. However, it should be noted that our evaluation period contains only the first 10 months of 2015, in which as many as 20 studies were published. On average, there were 2.5 studies per year between 1981 and 2013. During the 1990s, there was an average of 1.7 publications on the AHP per year, which increased to 4.6 per year between 2000 and 2013. In 2014 and 2015 the average increased to the peak of 18.5 studies, although the last two months of 2015 are not included.

Included Studies by Year of Publication

Most studies were from Asia (29.75 %), followed by the US (25.62 %). Almost all studies published before 2000 were conducted in the US ( n  = 15). However, between 2000 and 2010, a larger proportion came from Asia ( n  = 8) and Europe ( n  = 7), although most were still from the US ( n  = 8). Since 2010, Asia ( n  = 26) and Europe ( n  = 17) have surpassed the number of publications in the US ( n  = 8).

Another important aspect of these studies is the number of hierarchical levels that they include. Therefore, the studies could include more than one hierarchy, so in some cases the number of studies did not sum up to 121. More than half of the studies (51 %) included three hierarchical levels, 23 % described their hierarchy with two levels, and 21 % used four levels. On average, the studies used 19.76 criteria throughout their hierarchal levels. At the second hierarchical level, 96 articles (78 %) included between 1 and 5 criteria (Fig.  4 ). At the third and fourth levels, most studies ( n  = 39 and n  = 16 or 45 and 47 %, respectively) used between 11 and 20 criteria. The number of studies with five hierarchical levels was quite small ( n  = 3). As expected, the number of criteria increases as the hierarchical level increases. The right bar in Fig.  4 shows the total number of criteria for all hierarchical levels per study.

Number of Criteria per Hierarchical Level

Following the method set forth by Hummel and Ijzerman [ 22 ], we divided the studies into five categories: development of clinical guidelines, healthcare management, government policy, shared decision making, and biomedical innovation. We classified 38 studies (31 %) as pertaining to the development of clinical guidelines or recommendations, 30 (25 %) to healthcare management, 26 (21 %) to government policy, 15 (12 %) to biomedical innovation, and12 (10 %) to shared decision making.

Detailed analysis of the current state of research

This subsection summarizes the results of our analyses of the articles published within the last 5 years (January 2010 to October 2015). We examine how the studies design their hierarchies and carry out their surveys, and which analytical steps they take. In doing so, we follow the steps for conducting an AHP shown in Fig.  1 .

Definition of decision goal, criteria, and alternatives

The first step in conducting an AHP is to define the decision goal and criteria that describe the goal at a lower hierarchical level. In order to do this, many studies relied on literature research [ 20 , 25 , 26 , 45 – 83 ]. In addition, many studies relied on expert interviews [ 20 , 45 – 49 , 51 , 54 , 56 – 58 , 61 , 66 – 71 , 74 , 75 , 77 , 78 , 81 – 97 ] or focus groups [ 26 , 51 , 69 , 82 , 87 , 98 ]. Almost all of the studies defined their criteria by analyzing more than one source of information, although five publications did not explain their process for this step [ 99 – 103 ]. Some authors defined the criteria according to standards or established guidelines [ 25 , 50 , 52 , 59 , 80 , 84 , 92 , 93 , 104 – 108 ] or even from previous study results [ 25 , 47 , 62 , 68 , 69 , 71 , 72 , 81 ]. Still other authors relied on their own expertise [ 64 , 73 , 107 , 109 , 110 ].

Judgment through pairwise comparisons

The sample sizes varied between one author who judged the AHP for himself [ 73 , 107 – 109 ] to 1,283 participants [ 55 ]. In total, 50 of the 69 articles reported the number of participants in their AHP studies. The mean number of participants in these studies was about 109. Depending on the studies’ goal, the participants belonged to the following groups: hospital employees [ 49 , 92 ], patients [ 25 , 47 , 55 , 59 , 60 , 64 , 69 , 72 , 75 , 82 , 95 , 98 ], public/consumers [ 52 , 70 , 103 ], doctors or specialists [ 26 , 71 , 72 , 74 , 79 , 81 , 83 , 93 , 94 , 96 , 97 , 99 , 110 ], medical students [ 80 ] or teachers [ 77 ], biomedical engineers [ 94 ], technical experts [ 93 ], managers [ 93 ], administrators [ 20 ], and stakeholders [ 75 ]. Of the studies, 44 interviewed experts [ 20 , 26 , 45 , 46 , 48 – 51 , 54 , 56 – 58 , 61 , 62 , 66 – 68 , 71 , 74 , 76 – 79 , 81 , 83 – 94 , 96 , 97 , 99 , 104 – 107 , 110 ], 11 studies surveyed consumers or patients [ 25 , 47 , 52 , 55 , 59 , 60 , 69 , 70 , 82 , 98 , 103 ], and four studies included both [ 64 , 72 , 75 , 95 ]. However, six authors did not mention who answered the AHP questions [ 53 , 63 , 65 , 100 – 102 ].

Next, we considered whether the AHP was applied at individual or group level. Most of the studies questioned their participants individually [ 20 , 25 , 26 , 47 , 55 , 56 , 59 , 61 , 62 , 64 , 66 , 69 – 71 , 74 , 75 , 77 , 79 – 83 , 87 – 90 , 94 , 97 – 99 , 103 , 104 , 109 – 111 ]. On the other hand, only six articles mentioned group decisions [ 46 , 49 , 72 , 84 , 92 , 96 ]. Five studies conducted individual judgments as well as group decisions [ 51 , 60 , 86 , 93 , 95 ]. The remaining 23 articles did not describe the judgment, or they had only one person who answered.

In addition, there were differences in the applied scales for the pairwise comparisons. As explained in Subsection 1.1, the original scale implemented by Saaty ranges from nine (or 1/9) to one to nine. This scale was adopted by 37 of the articles in our sample [ 25 , 45 , 46 , 50 – 52 , 54 – 57 , 60 – 62 , 66 , 71 – 73 , 75 , 79 , 80 , 83 , 84 , 86 – 89 , 91 , 92 , 94 , 95 , 97 , 98 , 102 , 103 , 107 – 109 , 111 ]. Other studies used ranges between 1 and 4 [ 20 , 59 ], 1 and 5 [ 67 , 70 , 106 ], 5 and 1 and 5 [ 26 , 81 , 90 , 110 ], 6 and 1 and 6 [ 99 ],1 and 7 [ 47 ],1 and 9 [ 58 , 77 , 96 ], and 1 and 11 [ 74 ]. The remainder of the studies did not provide information about their scale [ 48 , 49 , 53 , 63 – 65 , 68 , 69 , 76 , 78 , 82 , 85 , 93 , 104 ].

Furthermore, there were major differences in how the surveys were conducted. Once again, not all of the authors discussed their process in detail, but those that did so used online questionnaires [ 20 , 47 , 51 , 55 , 58 , 70 , 74 , 75 , 81 – 83 , 111 ] (emailed) questionnaires [ 26 , 59 , 64 , 66 , 71 , 77 , 79 , 80 , 86 , 91 , 94 , 95 , 104 , 110 ], face-to-face interviews [ 25 , 45 , 87 , 90 , 98 ], group discussions or workshops [ 49 , 60 , 64 , 72 , 84 , 86 , 92 , 93 , 96 ], or Delphi panel method [ 61 ].

Analysis and validation of results

Specific software can support the AHP design and further analyses. However, only 35 of the 69 studies (49.28 %) mentioned which software they used. The majority of the studies that reported software chose Expert Choice® (23.19 %), while others used such packages as Microsoft Excel [ 25 , 77 , 88 , 90 ], or IBM SPSS Statistics [ 45 , 53 , 80 , 99 , 104 ]. In the last 5 years, a more diverse range of software packages has been in use; in addition to the aforementioned packages, researchers have chosen Super Decisions TM or Crystal Xcelsius [ 73 , 107 ], or programmed their own software [ 20 ].

The detailed analysis showed that 22 out of the 69 studies did not state a CR. However, 31 studies used a CR of 0.1 [ 20 , 26 , 45 , 46 , 49 – 51 , 56 , 57 , 60 – 62 , 67 , 71 – 74 , 76 , 77 , 83 , 87 , 89 , 91 , 98 – 102 , 107 – 109 ], five studies widened the range to a CR of 0.15 [ 25 , 59 , 64 , 75 , 111 ], and three studies accepted a CR of 0.2 or less [ 70 , 81 , 97 ]. The remaining studies did not establish a threshold prior to measuring average CRs [ 55 , 80 ]. As a consequence of these consistency conditions, 14 of the studies reported the number of participants that must be excluded in order to meet their established threshold [ 47 , 55 , 59 , 61 , 63 , 70 – 72 , 75 , 78 , 81 , 98 , 99 , 104 ]. However, only a small proportion of the studies actually outlined a procedure for dealing with excessive inconsistency (i.e., a CR above the established threshold). Chen et al. [ 70 ] and Pecchia et al. [ 26 ] asked the participants to fill out their questionnaires again. Hummel et al. [ 94 ], Suner et al. [ 83 ], Velmurugan et al. [ 102 ], and Cancela et al. [ 51 ] asked the participants to check and revise their decisions. Chung et al. [ 71 ], Li et al. [ 77 ], and Pecchia et al. [ 81 ] excluded the inconsistent participants from their analyses. Hou et al. [ 67 ] wrote that, in this case, “the judgment matrix has to be modified and recalculated.” Page et al. [ 80 ] ran simulations in which they assumed that the inconsistent answers were, in fact, consistent in the first place.

Furthermore, we examined group decision making. Danner et al. [ 72 ], Lin et al. [ 91 ], Papadopoulos et al. [ 56 ], Reddy et al. [ 86 ], Shojaei et al. [ 87 ], Jaberidoost et al. [ 66 ], and Hsu et al. [ 90 ] explored this topic by taking the geometric mean of the individual weights. Hilgerink et al. [ 93 ] and Hummel et al. [ 94 ] summarized the individual judgments with geometric means, and then, calculated the group weights. Conversely, other studies only averaged the group judgments [ 75 , 95 ]. Olivieri et al. [ 79 ] presented two AHPs; in the first, they calculated geometric means for the ranks and in the second, they calculated the inter-participant, standardized, geometric means of the weights as well as the inter-participant means. Perseghin et al. [ 96 ], Uzoka et al. [ 97 ], and Kuruoglu et al. [ 98 ] aggregated the participants’ judgments according to the median, and then, calculated the weights. By contrast, Taghipour et al. [ 49 ] constructed the group judgments by using weighted means. Unfortunately, 40 of the studies did not describe their weight calculations in detail [ 20 , 45 – 48 , 50 – 55 , 57 , 58 , 61 – 65 , 67 – 70 , 73 , 74 , 77 – 79 , 82 , 85 , 88 , 89 , 96 , 99 – 101 , 103 , 104 , 106 , 107 , 110 ]. However, 39 authors mentioned that they used the EV [ 25 , 26 , 45 – 47 , 49 , 50 , 55 – 57 , 59 , 60 , 62 , 65 , 66 , 71 , 72 , 75 , 76 , 80 , 81 , 83 , 86 – 95 , 97 , 100 , 102 , 104 , 105 , 108 , 109 ].

Very few of the studies ( n  = 14) examined the robustness of the weights [ 46 , 53 , 56 , 73 , 76 , 78 , 80 , 82 , 86 , 93 , 100 , 101 , 105 , 107 ]. Diaz-Ledezma et al. [ 107 ] and Diaz-Ledezma and Parvizi [ 73 ] referred to Erkut and Tarimcilar [ 40 ], who introduced sensitivity analysis for the AHP. Hilgerink et al. [ 93 ] factored in uncertainty regarding the included criteria by asking participants to rate the sensitivity and specificity of the pairwise judgments on a three-point scale; this yielded negative, average, and positive scenarios for the overall priorities. The other studies did not mention efforts to account for uncertainty. Further studies conducted their sensitivity analyses with the graphics provided in Expert Choice ® [ 100 , 101 ].

This subsection presents the most relevant aspects of conducting AHP, and thereby, reveals a high proportion of missing information from the literature. However, we summarize these facts in Subsection 3.2 and evaluate the number of reported aspects.

Evaluation of reporting

In a final step, we evaluated the reporting of the studies (see Subsection 2.2). Therefore, we suggested ten criteria that the authors should address in their articles. Most of the aspects are described in Subsection 3.1, and so, we focus on the number of reported elements for evaluating the studies in this section. We evaluated the studies published between 2010 and 2015 (until the 27th of October) and the detailed table can be found in Appendix : Table 1. In addition, we summarized the most important aspects from the table in the following graphs.

Figure  5 shows that all of the studies ( n  = 69) reported their decision goal and their criteria in their publications. However, several studies did not describe their interview process and did not mention which software they used. Particularly, only 15 out of 69 studies reported that they conducted sensitivity analysis.

Number of Studies by the Reported Criteria

The minimum number of reported criteria is one, namely, the study of Hsu et al. [ 63 ]. They described the aim of the study (assessment of oral phosphodiesterase type 5 inhibitors for treatment decisions of erectile dysfunction) and the hierarchy for the AHP but said nothing about the methods or study process. The studies that reported the highest number of ten criteria were published by Page [ 80 ] and Maruthur et al. [ 111 ]. The mean of the reported elements is 6.75, whereas only 12 out of 69 studies (17.39 %) reported less than half of the criteria.

The next figure demonstrates the results from our evaluation of reporting quality (Fig.  6 ). This figure shows the results from our evaluation regarding the reporting quality of all publications between 2010 and 2015. The highest number of studies reached seven or eight points in the evaluation. Only a small number of studies ( n  = 2) reported one or two aspects required. However, two publications also reported all of the criteria. The mean of reported criteria is 6.75.

Evaluation Results for Reporting Quality

Furthermore, we divided the publications into two time periods because we wanted to examine whether the reporting quality has changed (not shown graphically). Therefore, we took the studies published between 2010 and 2013 and compared them with the recent state of research since 2014 (the peak of published studies seen in Fig.  3 ). In the last 2 years, five studies got nine points in comparison to only three studies in the early time period. Indeed, two publications from the last 2 years only reached one or two points compared to no publications between 2010 and 2013. As the mean of the reported criteria is 6.88 for the early period and 6.65 for the last 2 years. Apparently we do not see the expected increase of reporting quality.

As seen from the review, in the last 10 years (and particularly in the last 2 years), there has been a clear upward trend in the number of publications that apply the AHP to healthcare. One reason for this could be the increasing acceptance and the discussion about integration of this method into policy decision processes. For example, the IQWiG in Germany suggests the AHP in decision making regarding reimbursement as one appropriate method [ 8 ]. Currently, the development of clinical guidelines is the most popular subject for AHP studies, followed by healthcare management decisions.

In the first step, the authors have to decompose their research question and set up a hierarchy for the AHP. Therefore, we have seen that most of the authors rely on literature research and expert opinions. This proceeding could carry the risk to not including further important criteria that have not been covered before but that are important for the overall problem and for the complete hierarchy. In particular, the perspective of the participants (in contrast to previous research) could require new criteria for the AHP.

The review showed wide fields for choosing participants in the AHP studies, even though a large portion of papers described their samples as experts or potential consumers of goods or services in question. Sample size was an important factor in these studies, for while there is no precise rule, there is general consensus that the AHP does not require a particularly large sample [ 23 ]. Consequently, it should be noted that the results are not necessarily representative. The number of participants ranged from 1 (a single author who judged the AHP for himself) to almost 1,300 with the mean being about 109. This wide range could influence the studies’ results. The evaluation of reporting in Subsection 3.2 examined satisfactory reporting of the participants in most of the papers. However, common rules for the process should be developed and several of its aspects improved upon. For instance, future research should develop a standardized method for calculating the sample size. Furthermore, the identification of the correct study sample is imperative in order to answer the studies’ research question properly.

In some cases, the participants were invited to revise their answers in case of inconsistency, and thereby, participants could be unsettled and biased. However, inconsistent judging could also be an indicator of overstraining the participants. Furthermore, most of these studies carried out the AHP on an individual basis, whereas only four authors mentioned group decisions. This was an unexpected finding because the AHP was introduced initially to study group decisions. However, our evaluation of the studies’ reporting showed that only six authors did not mention whether they had conducted group or individual decisions. Moreover, the aggregation of the AHP results from the individual level to a group did not present a uniform set of results. The advantage of group consensus is that it allows for the discussion of pairwise comparisons, which, in turn, improves participants’ understanding of the problem and criteria, and thereby, participants answer less inconsistently. This is because, on the one hand, they discuss their decisions before they set their judgments, but on the other hand, it may be because of the consensus or average extreme judgments being compensated by the group. Thus, the quality of the decision, seen as consistency, is improved [ 112 ]. Otherwise, the composition of the group would be a highly influential factor in the process of reaching consensus. This is because individuals within the group could have opposite priorities or else could be unwilling to discuss their positions. In this case, it would not be possible to reach a unanimous vote. Thus, another alternative is to aggregate the individual judgments [ 113 ]. In order to do this, one may take the geometric mean or median of either the individual judgments or the individual weights. One prerequisite is that the reciprocal of the aggregated values must correspond to the individual reciprocal values [ 28 ]; this can be achieved only by taking the geometric mean [ 113 ]. Unfortunately, only 29 of the 69 studies describe their exact processes for calculating the weights, but 39 reported using the EV in some way.

Recently, researchers have paid some attention to whether the results of these studies are robust. Despite the fact that sensitivity analyses could offer more information on the problem of rank reversal as well as the interpretation of the outcome [ 23 ], only 14 out of the 69 studies that we examine reported conducting such tests [ 73 , 76 , 78 , 82 , 93 , 107 ]. However, sensitivity analysis for AHP is relevant only when alternatives are included in the hierarchy. Consequently, 25 of 37 studies from our analysis missed reporting sensitivity analyses, as shown in Appendix : Table 2. One study without alternatives in the hierarchy suggested the use of standard deviations for weights [ 80 ]. The other sensitivity analysis presented in Subsection 1.1 requires a firm understanding of matrix algebra, does not yield fast or easy solutions, and is not supported by any software package. Although Expert Choice® provides the opportunity for sensitivity analysis, it offers only graphical simulation of one weight at the first hierarchical level [ 31 ]. Despite these challenges, sensitivity analyses remain vitally important as they allow researchers to assess the robustness of judgments, identify critical criteria or alternatives, find consensus through a range of judgments, and investigate different scenarios that support the decision [ 31 ]. Recently, Broekhuizen et al. have taken a further step concerning sensitivity analysis by providing an overview of dealing with uncertainty in multi-criteria decision making [ 114 ]. The results from sensitivity analysis can indicate potential rank reversal. The long-running dispute of rank reversal in AHP raised the question of “[…] the validity of AHP and the legitimacy of rank reversal” [ 42 ]. Wang et al. [ 42 ] argued that rank reversal is not only a phenomenon in the AHP but also in other decision making approaches. Saaty stated that the relative measurement of alternatives in the AHP implied by definition that all included alternatives were relevant, in contrast to utility theory that could face rank reversal problems [ 115 ]. Apart from these fundamental questions, several authors have suggested modifications to the AHP to overcome the problem of rank reversal [ 116 ].

Our evaluation of the reported criteria emphasizes the need to increase the number of given information in AHP studies. In general, authors should improve reporting on methodology, which is essential for comprehending and reproducing other authors’ results. This would serve to facilitate other researchers’ evaluations of study quality. In our opinion, two central explanations are possible for the current underreporting in the literature. First, the AHP, being fairly new, has few precisely formulated methodological rules. Second, what rules there are do not hold in practice. The latter observation also encompasses cases in which the AHP was too difficult for participants, either because of the formulations of the criteria or because of the method itself. It can be concluded that further research, in particular, methodological research, is needed in this field.

Although this study is based on systematic literature research and transparent evaluation criteria, there are a number of limitations that bear mentioning. As we primarily conducted our research on the Pubmed and Web of Science databases, it is possible that we did not include all relevant articles from other databases, even though we conducted a manual research. In addition, not all studies reported their procedures and methodologies in detail; therefore, the resulting statements in this review and the evaluation of the studies’ reporting could be biased, as we were restricted to available information. We are unable to make statements about the appropriateness of the evaluated content, like the sample size. By contrast, our evaluation criteria considered only whether a point was mentioned. Furthermore, the evaluation of reporting relied on the CONSORT and PRISMA Statements in order to develop criteria for the AHP. These statements suggest evaluation criteria for RCTs and systematic literature reviews, thus it could be criticized that we apply them to the subjective method of the AHP. The importance of each criterion can be criticized and our overall evaluation provides only an indication of the studies’ reporting with respect to informational content—not the quality. Moreover, we summarized the articles’ procedures but were unable to convey their results without some adaptions and generalizations; some aspects of the AHP must be adapted to suit the situation.

We found that there is a pressing need to develop methodological standards for the AHP; otherwise, discrepancies in methodology could bias studies’ results. In particular, future research should establish a standard procedure for aggregating individual data, specifically, a standard for using the geometric mean versus the arithmetic mean and aggregating judgments or priorities. We should place special emphasis on finding practical sensitivity analysis to address the criticisms regarding rank reversal due to changed judgments. In addition, suggestions are necessary for reporting the robustness of weights for AHPs that do not include alternatives.

Besides the methodological aspects of the AHP, we should also think about the topic that is researched. We carved out that the AHP is based on the hierarchical structure and the criteria that are included. If the author uses improper assumptions, he will find biased results. Therefore, the AHP hierarchy should not only base on one source of information but also on a combination of different methods (e.g. literature research and expert interview). Hence, further research is required about how to determine the interviewees, what should be done with inconsistent answers, and how the outcomes and the stability of the results should be presented. In the future, we need new insights as to which target groups can best handle the challenges of the AHP. These challenges are mainly consistent answering, preventing overstraining by using adequate numbers of pairwise comparisons, and deciding between group and individual AHP. Therefore, researchers should investigate specific groups, like elderly people, healthy people, and patients with different diseases or disabilities.

In our study, we analyzed whether authors reported important aspects of the AHP in their studies. This could be a first step to evaluate the quality of studies applying AHP in healthcare. In addition, guidelines should be formulated as to which statistics should be reported and how to conduct high-quality AHPs. As mentioned before, Bridges et al. published a checklist that contains recommendations for conducting conjoint analyses on healthcare topics on behalf of the International Society for Pharmacoeconomics and Outcomes Research (ISPOR) group [ 9 ]. Besides aspects for study presentation, it suggests criteria for evaluating the choice of attributes and the appropriateness of the method for the research question. Still further, we should take the current criticisms of the AHP into consideration so that we can find solutions to address them.

This systematic literature review shows a heterogeneous picture for application of the AHP in health economics research. It is likely that interest in the AHP will rise in the future, particularly in its application to health economic evaluations, the weighing of therapy outcomes, and benefit assessments. In this context, the AHP method could support decision making regarding reimbursement of pharmaceuticals. This is largely owing to its ability to translate complex questions into stepwise comparisons at different hierarchical levels. In these hierarchies, both quantitative and qualitative criteria can be compared, which provides a more accurate representation of real-world healthcare issues. Therefore, it should be used for complex decision problems that can completely be decomposed into a hierarchical structure. Thus, patients could apply the AHP to clarify their priorities. The patients could also benefit from these structured decisions in conversations with their physicians. The second important point is to figure out by researches which are the appropriate participants that are able to judge this research problem reliably.

Acknowledgements

The Center for Health Economics Research Hannover (CHERH) is founded by the Federal Ministry of Education and Research.

Abbreviations

Key words for systematic literature review

P patients, C potential consumers, E Experts, n/a not applicable, ind individual, g group, online online or web-based questionnaire, f2f face-to-face interview, lit literature, quest questionnaire (not further defined), ppq paper-pencil questionnaire, email mailed questionnaire, CR accepted consistency ratio, EV Eigenvector method, GA group average, GGM group geometric mean, WM weighted means, AN additive normalization method, alt alternatives included in the study, SD standard deviation

List of all included studies

Competing interests

The author(s) declare that they have no competing interests.

Authors’ contributions

KS carried out the analyses and drafted the manuscript. IA and IH participated in the review process and decision making process for identifying relevant articles. IA made substantial contributions to conception of the article. IH collected and prepared the data adequately for the manuscript. KD participated in selection process of papers and she revised the manuscript. JMS revised the manuscript for important intellectual content. All authors read and approved the final manuscript.

Contributor Information

Katharina Schmidt, Phone: +49 511 762 173 46, Email: ed.hrehc@sk , Email: ed.revonnah-inu.lbvi@sk .

Ines Aumann, Email: ed.hrehc@ai .

Ines Hollander, Email: [email protected] .

Kathrin Damm, Email: ed.hrehc@dk .

J.-Matthias Graf von der Schulenburg, Email: ed.hrehc@smj .

• Research article
• Open access
• Published: 24 December 2015

Applying the Analytic Hierarchy Process in healthcare research: A systematic literature review and evaluation of reporting

• Katharina Schmidt 1 ,
• Ines Aumann 1 ,
• Ines Hollander 2 ,
• Kathrin Damm 1 &
• J.-Matthias Graf von der Schulenburg 1  

BMC Medical Informatics and Decision Making volume  15 , Article number:  112 ( 2015 ) Cite this article

19k Accesses

106 Citations

2 Altmetric

Metrics details

The Analytic Hierarchy Process (AHP), developed by Saaty in the late 1970s, is one of the methods for multi-criteria decision making. The AHP disaggregates a complex decision problem into different hierarchical levels. The weight for each criterion and alternative are judged in pairwise comparisons and priorities are calculated by the Eigenvector method. The slowly increasing application of the AHP was the motivation for this study to explore the current state of its methodology in the healthcare context.

A systematic literature review was conducted by searching the Pubmed and Web of Science databases for articles with the following keywords in their titles or abstracts: “Analytic Hierarchy Process,” “Analytical Hierarchy Process,” “multi-criteria decision analysis,” “multiple criteria decision,” “stated preference,” and “pairwise comparison.” In addition, we developed reporting criteria to indicate whether the authors reported important aspects and evaluated the resulting studies’ reporting.

The systematic review resulted in 121 articles. The number of studies applying AHP has increased since 2005. Most studies were from Asia (almost 30 %), followed by the US (25.6 %). On average, the studies used 19.64 criteria throughout their hierarchical levels. Furthermore, we restricted a detailed analysis to those articles published within the last 5 years ( n  = 69). The mean of participants in these studies were 109, whereas we identified major differences in how the surveys were conducted. The evaluation of reporting showed that the mean of reported elements was about 6.75 out of 10. Thus, 12 out of 69 studies reported less than half of the criteria.

The AHP has been applied inconsistently in healthcare research. A minority of studies described all the relevant aspects. Thus, the statements in this review may be biased, as they are restricted to the information available in the papers. Hence, further research is required to discover who should be interviewed and how, how inconsistent answers should be dealt with, and how the outcome and stability of the results should be presented. In addition, we need new insights to determine which target group can best handle the challenges of the AHP.

Peer Review reports

The resources in health care systems are limited. Exacerbating this issue is the problem that many developed countries face, that is, the rising proportion of older, multimorbid patients, who serve to raise the cost of health care. Furthermore, innovations in medical care, such as equipment, pharmaceuticals, and treatment methods, are also driving up costs. German politicians have adopted new laws to manage the costs of pharmaceuticals, e.g. the Act on the Reform of the Market for Medicinal Products in 2011 (in German: AMNOG [ 1 ]). In this context, patient-relevant outcomes have drawn greater attention because the added benefit for patients determines the reimbursement price. But also, other countries are interested in reliable methods to measure benefits for patients, for example, to support Health Technology Assessments by patient preferences [ 2 , 3 ]. Therefore, while it is now important to measure the benefits and to prioritize the needs of patients, it will be even more so in the future. However, several studies have found a divergence in patients’ and physicians’ preferences or priorities regarding prevention and therapy (e.g. [ 4 – 6 ]). Thus, one mean of evaluating these preferences and bringing them into accord is to take the required perspective for the situation. In order to find appropriate methods for measuring the benefits and for prioritizing them, beside the established methods, new approaches of decision making tools are transferred from other fields of research, like the marketing sector. For all of these methods it is essential to measure the trade-off between attributes in multi-criteria decision situations for each participant or the group, and as such, adequate and understandable methods are essential.

Several methods are known for multi-criteria decision making in the field of health care, including value based methods, strategy based methods, and conjoint analyses [ 7 ]. In Germany, the Institute for Quality and Efficiency in Health Care (IQWiG) suggested two methods for multi-attribute decision making: Conjoint Analysis (CA) and the Analytic Hierarchy Process (AHP) [ 8 ]. Although they concluded that both methods are applicable for decision making, they were also confronted with methodological limitations. As the advantages and disadvantages of established methods like the CA have been discussed in a number of publications (e.g. [ 9 – 11 ]), the AHP method has received less attention. Therefore, we wanted to figure out whether the AHP method could become a good alternative in multi-criteria decision making.

Relevance and objective of the study

The Analytic Hierarchy Process (AHP) was developed by Saaty in the late 1970s and originally was applied to the marketing sector [ 12 , 13 ]. Dolan et al. were the first to apply this method to health economics research in 1989 [ 14 , 15 ]; since then, it has been accepted slowly as a method in the field of multi-criteria decision making in healthcare. Liberatore and Nydick described the importance of applying the AHP as follows: “Health care and medical decision making has been an early and on-going application area for the AHP” [ 16 ]. The AHP method was applied to different contexts, for example, the development of clinical guidelines [ 17 , 18 ] or biomedical innovations and technology development [ 19 , 20 ].

The increasing application of the AHP has been the motivation for this study to explore the current state of its methodology. The method is the basis for assessing the best instrument for each decision situation and reflecting each participant’s opinion correctly. A review provides an overview of published papers in this field. In line with De Bekker-Grob et al. [ 21 ], we provide a systematic review of the AHP. Therefore, an overview is given of the year of publication, country, and number of criteria used in the AHP (Section 3). In addition, Hummel and Ijzerman [ 22 ] analyzed the thematic field in which AHP is used. They identified the different areas of application (e.g., shared decision making, clinical guidelines, and healthcare management), number of criteria and alternatives, individual or group decisions, participants, and rating method. We focus on the methodological applications in the second step. In addition, the analyzed time horizon (2010–2015) should provide an update on Hummel and Ijzerman’s study and allow us to provide details of the most recent developments in the subject area. As in Mühlbacher’s overview [ 23 ], the field of application and the sample are inspected, although our focus remains on the current state of the research (the last 5 years) and the reporting of methodological aspects in the papers. In addition, the evaluation of studies’ reporting allows deeper insights. Therefore, we develop criteria for reporting the AHP method and determine to what extent the studies fulfill the criteria. We conclude by proposing recommended situations in which the AHP can be used.

AHP – a short introduction

As a short introduction into the method of AHP, we report the most important aspects here. We refer to detailed papers to provide deeper insights into specific methodological aspects.

The AHP disaggregates a complex decision problem into different hierarchical levels (see Saaty’s axioms for the AHP [ 24 ]). The application of an AHP is structured into six steps (see also Fig.  1 ), suggested by Dolan et al. [ 25 ] and Dolan [ 7 ], as follows: 1. define the decision goal, criteria, and alternatives, 2. rate the criteria in pairwise comparisons, 3. calculate the relative priority weights for the (sub-)criteria, 4. calculate the criteria’s global priority weights and combine the alternatives’ priorities, 5. control for inconsistency, and 6. perform sensitivity analysis.

Steps of the AHP (modeled after Dolan et al. [ 25 ] and Dolan [ 7 ]])

At the first hierarchical level, the aim of the study is defined followed by the main criteria, which can be divided further at lower levels into sub-criteria. If necessary, alternatives that contain specific combinations of characteristics can be arranged at the lowest level of the hierarchy. Although the AHP was introduced for group decisions, it may also be applied to single person decisions [ 26 ]. Pairwise comparisons at each hierarchical level present the judgments and they must be evaluated according to a scale developed by Saaty, which ranges from 9 to 1 to 9. If the alternatives consisted of subjective combinations of the criteria, the alternatives would be judged also with regard to each criterion. Saaty provided a detailed description of his scale and its intensities [ 12 ].

In order to analyze the interviews, the pairwise comparisons of (sub-)criteria at each level are displayed in ordered schemes (matrixes). An example is seen in Saaty ([ 24 ], p. 164). Only half of the matrix has to be filled in, as the other half is obtained from the reciprocal weights. The Eigenvector method (EV) is the most common means of calculating the priority vector, although other methods, such as additive normalization, weighted least-squares, logarithmic least-squares, logarithmic goal programming, and fuzzy preference programming methods, yield comparable results [ 27 ]. The EV relies on the matrix’s principle eigenvalue, which results from a process of repeated squaring and normalization (for more information, see Srdjevic [ 27 ] or Saaty [ 12 ]). The resulting local weights describe the relative priorities in relation to their parent criterion. The local weights form the global weights for the criteria through multiplication with the local weights from their parent criteria [ 24 ]. Thereby, global weights for criteria show the importance of each criterion in the overall context of the hierarchy. The priorities for the alternatives of the AHP are calculated by the sum of the particular local and global weights for each alternative [ 23 ]. For detailed information and examples concerning the calculations, see Saaty [ 28 ].

The aggregation of the individual judgments or priorities is fundamental to the outcome of the study. The first option is to have the group of participants vote by finding consensus. Another alternative is to aggregate the individual judgments. Still further, the literature suggests finding the geometric mean [ 29 ] or arithmetic mean [ 30 ]. In addition, the timing of calculating the average affects the results [ 30 ], specifically, the average of participants’ judgments or the average of participants’ global weights. Yet another option is to give special weight to one participant’s decision on the basis of that participant being an expert in the field or holding an exceptional position within the group [ 30 ]. The consistency ratio (CR) measures the uniformity of a respondent’s answers to the AHP questions. Saaty [ 24 ] describes the calculation of the CR in detail. The CR can also be calculated for a group of respondents.

Although the AHP has been applied to a variety of topics within the healthcare field, the sensitivity analyses on hierarchical decision making has received little investigation [ 31 ]. It should be noted that there are two distinct types of sensitivity analysis, that of judgments and that of priorities [ 32 ]. The former has been explained and tested by Arbel [ 33 ], Moreno-Jimenez and Vargas [ 34 ], and Sugihara and Tanaka [ 35 ]. They determined the judgments’ upper and lower bounds and articulated the preferences through preference structures. Other approaches originate from Moreno-Jimenez and Vargas [ 34 ], Triantaphyllou and Sánchez [ 36 ], Sowlati et al. [ 37 ], Masuda [ 38 ], and Huang [ 39 ]. Erkut and Tarimcilar [ 40 ] provided “a collection of practical tools for a potentially powerful sensitivity analysis in the AHP”. In addition, Altuzarra et al. [ 41 ] proposed a method for determining the stability of group decisions. If the AHP includes alternatives, the sensitivity analysis could show the effect of varying weights on the alternatives’ rankings [ 23 ]. Therefore, potential rank reversal of alternatives can be simulated. Rank reversal occurs when adding or deleting an (irrelevant) alternative leads to a shift in the previous alternatives’ ranking order [ 42 ].

This chapter is divided into two parts to introduce the methods used in this paper. The first part describes the method of the systematic review, which includes the key words and a flow chart. Further, in chapter 2.2, we describe our evaluation of reporting quality for the included studies.

• Systematic literature review

The basis of this review is a systematic literature research on the Pubmed and Web of Science databases (date of research: 10/27/2015). As we focused our research question on healthcare, we did not include further databases in the other scientific fields. We searched both databases for articles with the following keywords in their titles or abstracts: “Analytic Hierarchy Process,” “Analytical Hierarchy Process,” “multi-criteria decision analysis,” “multiple criteria decision,” “stated preference,” and “pairwise comparison.” We provided the search strategy in Appendix : Table 1. It was technically not possible to search Web of Science for keywords in the abstracts. We refined the search by including only articles written in German or English and those associated with healthcare. Two independent reviewers evaluated the titles and abstracts of the resulting studies. Therefore, the criterion for inclusion was that the article is the primary source and the study used the AHP method within the healthcare setting. Additionally, we conducted a manual search to find further articles not included in the aforementioned databases. Thereafter, the two reviewers screened the full texts of the remaining articles and discussed whether to include them in the review. After reaching consensus, the important information was summarized in a table (not shown). Apart from common information, like the author, title, publication year, country, and journal, we extracted additional information regarding the study’s aim, source of criteria identification, hierarchy design, form of implementation, and analytical steps in order to conduct our analysis. The results are described in Section 3 for the entire period and in detail for the last 5 years in Subsection 3.1. The first step should give a short overview of all studies that were conducted with AHP in health care. In the second step, we reported the current state of research in more detail.

Evaluation of reporting quality

The papers identified from the last 5 years resulting from the systematic review were evaluated with regard to their reporting quality. Because there was no set standard by which to judge the AHP’s methodological issues, the evaluation of the studies’ quality was quite challenging. The before mentioned studies by De Bekker-Grob et al. [ 21 ], Hummel and Ijzerman [ 22 ], and Mühlbacher et al. [ 23 ] did not report quality criteria. However, the Consolidated Standards of Reporting Trials (CONSORT) Statement for randomized controlled trials [ 43 ] and the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) Statement [ 44 ] may provide some direction by providing checklists for transparent and complete reporting. The reason why authors should report specific aspects is the traceability of the study. Some criteria from the CONSORT Statement could be transferred to AHP studies: sample size, participants (eligibility criteria), trial designs, and statistical methods. In the case of the AHP method, the latter criterion consists of the CR, the method used to calculate the weights, the statistical software, and sensitivity analyses. Another checklist item is the description of the intervention. Transferred to the AHP method, authors should provide information about the interview process. Besides, another guideline for good research practices is published by Bridges et al. [ 9 ]. They provide a detailed checklist that is specific for conducting conjoint analyses. Since it suggests quality aspects only for those kinds of studies, the checklist cannot be used directly for our evaluation. However, we summarized the recommendations from the different fields and we obtained a simplified measurement of reporting by counting the elements that were included in the studies. Therefore, we evaluated whether the authors mentioned aspects for the following elements in their papers:

Decision goal, criteria (and if necessary alternatives)

Number of participants

Type of participants (patients, potential consumers, or experts)

Decision making by group or individually

Scale for pairwise comparisons

Interview process (face to face, email, questionnaire, judgments based on literature)

Calculation of weights

Sensitivity analysis

The last criterion was valid only for studies including alternatives. Thus, for the other papers without alternatives, we could determine only whether descriptive statistics (e.g., standard deviation, SD and confidence intervals, CI) were reported for the judgments or weights. We calculated the sum of all reported aspects for each study and present the results in Appendix : Table 2 and we show charts in Subsection 3.2. Nevertheless, we could not evaluate the content of each of the abovementioned criteria but only whether the criteria were reported in the study.

The search in Pubmed yielded to 1,956 articles and the search in Web of Science yielded to 4,829 articles, as Fig.  2 shows. Furthermore, 44 additional records were found via manual search. By screening titles and abstracts, we limited the sample to 246 articles (we excluded a total of 6,485 articles based on language or irrelevance to healthcare and we found 54 duplicates). Thereafter, we examined the full articles in order to determine whether they apply AHP to the field of healthcare. An additional 125 papers were excluded because they were not original studies or they used other stated preference methods (e.g., discrete choice experiment). In total, this process yielded to 121 relevant studies; the Appendix : Table 3 provides a complete list. We provide a brief overview of these studies to show how many studies have been published in this field and in which context the authors used the AHP. In addition, the overview presents the development and the relevance to the AHP method. In order to explore the current state of the literature, we limited the body of our research to articles published within the last 5 years. This restriction reduced the number of studies to 69. The detailed analysis of these studies’ methodologies made it necessary to reduce the number of articles.

Flow Chart of the Systematic Literature Review

For a first overview, we briefly summarized the key factors of all of the relevant articles ( n  = 121), such as their publication year, country, number of attributes, and levels.

The earliest study to use the AHP was published in 1981, but the AHP has become increasingly popular since 2005 (see also Fig.  3 ). The 2 years with the greatest number of studies published on the subject were 2011 and 2012 with nine each. However, it should be noted that our evaluation period contains only the first 10 months of 2015, in which as many as 20 studies were published. On average, there were 2.5 studies per year between 1981 and 2013. During the 1990s, there was an average of 1.7 publications on the AHP per year, which increased to 4.6 per year between 2000 and 2013. In 2014 and 2015 the average increased to the peak of 18.5 studies, although the last two months of 2015 are not included.

Included Studies by Year of Publication

Most studies were from Asia (29.75 %), followed by the US (25.62 %). Almost all studies published before 2000 were conducted in the US ( n  = 15). However, between 2000 and 2010, a larger proportion came from Asia ( n  = 8) and Europe ( n  = 7), although most were still from the US ( n  = 8). Since 2010, Asia ( n  = 26) and Europe ( n  = 17) have surpassed the number of publications in the US ( n  = 8).

Another important aspect of these studies is the number of hierarchical levels that they include. Therefore, the studies could include more than one hierarchy, so in some cases the number of studies did not sum up to 121. More than half of the studies (51 %) included three hierarchical levels, 23 % described their hierarchy with two levels, and 21 % used four levels. On average, the studies used 19.76 criteria throughout their hierarchal levels. At the second hierarchical level, 96 articles (78 %) included between 1 and 5 criteria (Fig.  4 ). At the third and fourth levels, most studies ( n  = 39 and n  = 16 or 45 and 47 %, respectively) used between 11 and 20 criteria. The number of studies with five hierarchical levels was quite small ( n  = 3). As expected, the number of criteria increases as the hierarchical level increases. The right bar in Fig.  4 shows the total number of criteria for all hierarchical levels per study.

Number of Criteria per Hierarchical Level

Following the method set forth by Hummel and Ijzerman [ 22 ], we divided the studies into five categories: development of clinical guidelines, healthcare management, government policy, shared decision making, and biomedical innovation. We classified 38 studies (31 %) as pertaining to the development of clinical guidelines or recommendations, 30 (25 %) to healthcare management, 26 (21 %) to government policy, 15 (12 %) to biomedical innovation, and12 (10 %) to shared decision making.

Detailed analysis of the current state of research

This subsection summarizes the results of our analyses of the articles published within the last 5 years (January 2010 to October 2015). We examine how the studies design their hierarchies and carry out their surveys, and which analytical steps they take. In doing so, we follow the steps for conducting an AHP shown in Fig.  1 .

Definition of decision goal, criteria, and alternatives

The first step in conducting an AHP is to define the decision goal and criteria that describe the goal at a lower hierarchical level. In order to do this, many studies relied on literature research [ 20 , 25 , 26 , 45 – 83 ]. In addition, many studies relied on expert interviews [ 20 , 45 – 49 , 51 , 54 , 56 – 58 , 61 , 66 – 71 , 74 , 75 , 77 , 78 , 81 – 97 ] or focus groups [ 26 , 51 , 69 , 82 , 87 , 98 ]. Almost all of the studies defined their criteria by analyzing more than one source of information, although five publications did not explain their process for this step [ 99 – 103 ]. Some authors defined the criteria according to standards or established guidelines [ 25 , 50 , 52 , 59 , 80 , 84 , 92 , 93 , 104 – 108 ] or even from previous study results [ 25 , 47 , 62 , 68 , 69 , 71 , 72 , 81 ]. Still other authors relied on their own expertise [ 64 , 73 , 107 , 109 , 110 ].

Judgment through pairwise comparisons

The sample sizes varied between one author who judged the AHP for himself [ 73 , 107 – 109 ] to 1,283 participants [ 55 ]. In total, 50 of the 69 articles reported the number of participants in their AHP studies. The mean number of participants in these studies was about 109. Depending on the studies’ goal, the participants belonged to the following groups: hospital employees [ 49 , 92 ], patients [ 25 , 47 , 55 , 59 , 60 , 64 , 69 , 72 , 75 , 82 , 95 , 98 ], public/consumers [ 52 , 70 , 103 ], doctors or specialists [ 26 , 71 , 72 , 74 , 79 , 81 , 83 , 93 , 94 , 96 , 97 , 99 , 110 ], medical students [ 80 ] or teachers [ 77 ], biomedical engineers [ 94 ], technical experts [ 93 ], managers [ 93 ], administrators [ 20 ], and stakeholders [ 75 ]. Of the studies, 44 interviewed experts [ 20 , 26 , 45 , 46 , 48 – 51 , 54 , 56 – 58 , 61 , 62 , 66 – 68 , 71 , 74 , 76 – 79 , 81 , 83 – 94 , 96 , 97 , 99 , 104 – 107 , 110 ], 11 studies surveyed consumers or patients [ 25 , 47 , 52 , 55 , 59 , 60 , 69 , 70 , 82 , 98 , 103 ], and four studies included both [ 64 , 72 , 75 , 95 ]. However, six authors did not mention who answered the AHP questions [ 53 , 63 , 65 , 100 – 102 ].

Next, we considered whether the AHP was applied at individual or group level. Most of the studies questioned their participants individually [ 20 , 25 , 26 , 47 , 55 , 56 , 59 , 61 , 62 , 64 , 66 , 69 – 71 , 74 , 75 , 77 , 79 – 83 , 87 – 90 , 94 , 97 – 99 , 103 , 104 , 109 – 111 ]. On the other hand, only six articles mentioned group decisions [ 46 , 49 , 72 , 84 , 92 , 96 ]. Five studies conducted individual judgments as well as group decisions [ 51 , 60 , 86 , 93 , 95 ]. The remaining 23 articles did not describe the judgment, or they had only one person who answered.

In addition, there were differences in the applied scales for the pairwise comparisons. As explained in Subsection 1.1, the original scale implemented by Saaty ranges from nine (or 1/9) to one to nine. This scale was adopted by 37 of the articles in our sample [ 25 , 45 , 46 , 50 – 52 , 54 – 57 , 60 – 62 , 66 , 71 – 73 , 75 , 79 , 80 , 83 , 84 , 86 – 89 , 91 , 92 , 94 , 95 , 97 , 98 , 102 , 103 , 107 – 109 , 111 ]. Other studies used ranges between 1 and 4 [ 20 , 59 ], 1 and 5 [ 67 , 70 , 106 ], 5 and 1 and 5 [ 26 , 81 , 90 , 110 ], 6 and 1 and 6 [ 99 ],1 and 7 [ 47 ],1 and 9 [ 58 , 77 , 96 ], and 1 and 11 [ 74 ]. The remainder of the studies did not provide information about their scale [ 48 , 49 , 53 , 63 – 65 , 68 , 69 , 76 , 78 , 82 , 85 , 93 , 104 ].

Furthermore, there were major differences in how the surveys were conducted. Once again, not all of the authors discussed their process in detail, but those that did so used online questionnaires [ 20 , 47 , 51 , 55 , 58 , 70 , 74 , 75 , 81 – 83 , 111 ] (emailed) questionnaires [ 26 , 59 , 64 , 66 , 71 , 77 , 79 , 80 , 86 , 91 , 94 , 95 , 104 , 110 ], face-to-face interviews [ 25 , 45 , 87 , 90 , 98 ], group discussions or workshops [ 49 , 60 , 64 , 72 , 84 , 86 , 92 , 93 , 96 ], or Delphi panel method [ 61 ].

Analysis and validation of results

Specific software can support the AHP design and further analyses. However, only 35 of the 69 studies (49.28 %) mentioned which software they used. The majority of the studies that reported software chose Expert Choice® (23.19 %), while others used such packages as Microsoft Excel [ 25 , 77 , 88 , 90 ], or IBM SPSS Statistics [ 45 , 53 , 80 , 99 , 104 ]. In the last 5 years, a more diverse range of software packages has been in use; in addition to the aforementioned packages, researchers have chosen Super Decisions TM or Crystal Xcelsius [ 73 , 107 ], or programmed their own software [ 20 ].

The detailed analysis showed that 22 out of the 69 studies did not state a CR. However, 31 studies used a CR of 0.1 [ 20 , 26 , 45 , 46 , 49 – 51 , 56 , 57 , 60 – 62 , 67 , 71 – 74 , 76 , 77 , 83 , 87 , 89 , 91 , 98 – 102 , 107 – 109 ], five studies widened the range to a CR of 0.15 [ 25 , 59 , 64 , 75 , 111 ], and three studies accepted a CR of 0.2 or less [ 70 , 81 , 97 ]. The remaining studies did not establish a threshold prior to measuring average CRs [ 55 , 80 ]. As a consequence of these consistency conditions, 14 of the studies reported the number of participants that must be excluded in order to meet their established threshold [ 47 , 55 , 59 , 61 , 63 , 70 – 72 , 75 , 78 , 81 , 98 , 99 , 104 ]. However, only a small proportion of the studies actually outlined a procedure for dealing with excessive inconsistency (i.e., a CR above the established threshold). Chen et al. [ 70 ] and Pecchia et al. [ 26 ] asked the participants to fill out their questionnaires again. Hummel et al. [ 94 ], Suner et al. [ 83 ], Velmurugan et al. [ 102 ], and Cancela et al. [ 51 ] asked the participants to check and revise their decisions. Chung et al. [ 71 ], Li et al. [ 77 ], and Pecchia et al. [ 81 ] excluded the inconsistent participants from their analyses. Hou et al. [ 67 ] wrote that, in this case, “the judgment matrix has to be modified and recalculated.” Page et al. [ 80 ] ran simulations in which they assumed that the inconsistent answers were, in fact, consistent in the first place.

Furthermore, we examined group decision making. Danner et al. [ 72 ], Lin et al. [ 91 ], Papadopoulos et al. [ 56 ], Reddy et al. [ 86 ], Shojaei et al. [ 87 ], Jaberidoost et al. [ 66 ], and Hsu et al. [ 90 ] explored this topic by taking the geometric mean of the individual weights. Hilgerink et al. [ 93 ] and Hummel et al. [ 94 ] summarized the individual judgments with geometric means, and then, calculated the group weights. Conversely, other studies only averaged the group judgments [ 75 , 95 ]. Olivieri et al. [ 79 ] presented two AHPs; in the first, they calculated geometric means for the ranks and in the second, they calculated the inter-participant, standardized, geometric means of the weights as well as the inter-participant means. Perseghin et al. [ 96 ], Uzoka et al. [ 97 ], and Kuruoglu et al. [ 98 ] aggregated the participants’ judgments according to the median, and then, calculated the weights. By contrast, Taghipour et al. [ 49 ] constructed the group judgments by using weighted means. Unfortunately, 40 of the studies did not describe their weight calculations in detail [ 20 , 45 – 48 , 50 – 55 , 57 , 58 , 61 – 65 , 67 – 70 , 73 , 74 , 77 – 79 , 82 , 85 , 88 , 89 , 96 , 99 – 101 , 103 , 104 , 106 , 107 , 110 ]. However, 39 authors mentioned that they used the EV [ 25 , 26 , 45 – 47 , 49 , 50 , 55 – 57 , 59 , 60 , 62 , 65 , 66 , 71 , 72 , 75 , 76 , 80 , 81 , 83 , 86 – 95 , 97 , 100 , 102 , 104 , 105 , 108 , 109 ].

Very few of the studies ( n  = 14) examined the robustness of the weights [ 46 , 53 , 56 , 73 , 76 , 78 , 80 , 82 , 86 , 93 , 100 , 101 , 105 , 107 ]. Diaz-Ledezma et al. [ 107 ] and Diaz-Ledezma and Parvizi [ 73 ] referred to Erkut and Tarimcilar [ 40 ], who introduced sensitivity analysis for the AHP. Hilgerink et al. [ 93 ] factored in uncertainty regarding the included criteria by asking participants to rate the sensitivity and specificity of the pairwise judgments on a three-point scale; this yielded negative, average, and positive scenarios for the overall priorities. The other studies did not mention efforts to account for uncertainty. Further studies conducted their sensitivity analyses with the graphics provided in Expert Choice ® [ 100 , 101 ].

This subsection presents the most relevant aspects of conducting AHP, and thereby, reveals a high proportion of missing information from the literature. However, we summarize these facts in Subsection 3.2 and evaluate the number of reported aspects.

Evaluation of reporting

In a final step, we evaluated the reporting of the studies (see Subsection 2.2). Therefore, we suggested ten criteria that the authors should address in their articles. Most of the aspects are described in Subsection 3.1, and so, we focus on the number of reported elements for evaluating the studies in this section. We evaluated the studies published between 2010 and 2015 (until the 27th of October) and the detailed table can be found in Appendix : Table 1. In addition, we summarized the most important aspects from the table in the following graphs.

Figure  5 shows that all of the studies ( n  = 69) reported their decision goal and their criteria in their publications. However, several studies did not describe their interview process and did not mention which software they used. Particularly, only 15 out of 69 studies reported that they conducted sensitivity analysis.

Number of Studies by the Reported Criteria

The minimum number of reported criteria is one, namely, the study of Hsu et al. [ 63 ]. They described the aim of the study (assessment of oral phosphodiesterase type 5 inhibitors for treatment decisions of erectile dysfunction) and the hierarchy for the AHP but said nothing about the methods or study process. The studies that reported the highest number of ten criteria were published by Page [ 80 ] and Maruthur et al. [ 111 ]. The mean of the reported elements is 6.75, whereas only 12 out of 69 studies (17.39 %) reported less than half of the criteria.

The next figure demonstrates the results from our evaluation of reporting quality (Fig.  6 ). This figure shows the results from our evaluation regarding the reporting quality of all publications between 2010 and 2015. The highest number of studies reached seven or eight points in the evaluation. Only a small number of studies ( n  = 2) reported one or two aspects required. However, two publications also reported all of the criteria. The mean of reported criteria is 6.75.

Evaluation Results for Reporting Quality

Furthermore, we divided the publications into two time periods because we wanted to examine whether the reporting quality has changed (not shown graphically). Therefore, we took the studies published between 2010 and 2013 and compared them with the recent state of research since 2014 (the peak of published studies seen in Fig.  3 ). In the last 2 years, five studies got nine points in comparison to only three studies in the early time period. Indeed, two publications from the last 2 years only reached one or two points compared to no publications between 2010 and 2013. As the mean of the reported criteria is 6.88 for the early period and 6.65 for the last 2 years. Apparently we do not see the expected increase of reporting quality.

As seen from the review, in the last 10 years (and particularly in the last 2 years), there has been a clear upward trend in the number of publications that apply the AHP to healthcare. One reason for this could be the increasing acceptance and the discussion about integration of this method into policy decision processes. For example, the IQWiG in Germany suggests the AHP in decision making regarding reimbursement as one appropriate method [ 8 ]. Currently, the development of clinical guidelines is the most popular subject for AHP studies, followed by healthcare management decisions.

In the first step, the authors have to decompose their research question and set up a hierarchy for the AHP. Therefore, we have seen that most of the authors rely on literature research and expert opinions. This proceeding could carry the risk to not including further important criteria that have not been covered before but that are important for the overall problem and for the complete hierarchy. In particular, the perspective of the participants (in contrast to previous research) could require new criteria for the AHP.

The review showed wide fields for choosing participants in the AHP studies, even though a large portion of papers described their samples as experts or potential consumers of goods or services in question. Sample size was an important factor in these studies, for while there is no precise rule, there is general consensus that the AHP does not require a particularly large sample [ 23 ]. Consequently, it should be noted that the results are not necessarily representative. The number of participants ranged from 1 (a single author who judged the AHP for himself) to almost 1,300 with the mean being about 109. This wide range could influence the studies’ results. The evaluation of reporting in Subsection 3.2 examined satisfactory reporting of the participants in most of the papers. However, common rules for the process should be developed and several of its aspects improved upon. For instance, future research should develop a standardized method for calculating the sample size. Furthermore, the identification of the correct study sample is imperative in order to answer the studies’ research question properly.

In some cases, the participants were invited to revise their answers in case of inconsistency, and thereby, participants could be unsettled and biased. However, inconsistent judging could also be an indicator of overstraining the participants. Furthermore, most of these studies carried out the AHP on an individual basis, whereas only four authors mentioned group decisions. This was an unexpected finding because the AHP was introduced initially to study group decisions. However, our evaluation of the studies’ reporting showed that only six authors did not mention whether they had conducted group or individual decisions. Moreover, the aggregation of the AHP results from the individual level to a group did not present a uniform set of results. The advantage of group consensus is that it allows for the discussion of pairwise comparisons, which, in turn, improves participants’ understanding of the problem and criteria, and thereby, participants answer less inconsistently. This is because, on the one hand, they discuss their decisions before they set their judgments, but on the other hand, it may be because of the consensus or average extreme judgments being compensated by the group. Thus, the quality of the decision, seen as consistency, is improved [ 112 ]. Otherwise, the composition of the group would be a highly influential factor in the process of reaching consensus. This is because individuals within the group could have opposite priorities or else could be unwilling to discuss their positions. In this case, it would not be possible to reach a unanimous vote. Thus, another alternative is to aggregate the individual judgments [ 113 ]. In order to do this, one may take the geometric mean or median of either the individual judgments or the individual weights. One prerequisite is that the reciprocal of the aggregated values must correspond to the individual reciprocal values [ 28 ]; this can be achieved only by taking the geometric mean [ 113 ]. Unfortunately, only 29 of the 69 studies describe their exact processes for calculating the weights, but 39 reported using the EV in some way.

Recently, researchers have paid some attention to whether the results of these studies are robust. Despite the fact that sensitivity analyses could offer more information on the problem of rank reversal as well as the interpretation of the outcome [ 23 ], only 14 out of the 69 studies that we examine reported conducting such tests [ 73 , 76 , 78 , 82 , 93 , 107 ]. However, sensitivity analysis for AHP is relevant only when alternatives are included in the hierarchy. Consequently, 25 of 37 studies from our analysis missed reporting sensitivity analyses, as shown in Appendix : Table 2. One study without alternatives in the hierarchy suggested the use of standard deviations for weights [ 80 ]. The other sensitivity analysis presented in Subsection 1.1 requires a firm understanding of matrix algebra, does not yield fast or easy solutions, and is not supported by any software package. Although Expert Choice® provides the opportunity for sensitivity analysis, it offers only graphical simulation of one weight at the first hierarchical level [ 31 ]. Despite these challenges, sensitivity analyses remain vitally important as they allow researchers to assess the robustness of judgments, identify critical criteria or alternatives, find consensus through a range of judgments, and investigate different scenarios that support the decision [ 31 ]. Recently, Broekhuizen et al. have taken a further step concerning sensitivity analysis by providing an overview of dealing with uncertainty in multi-criteria decision making [ 114 ]. The results from sensitivity analysis can indicate potential rank reversal. The long-running dispute of rank reversal in AHP raised the question of “[…] the validity of AHP and the legitimacy of rank reversal” [ 42 ]. Wang et al. [ 42 ] argued that rank reversal is not only a phenomenon in the AHP but also in other decision making approaches. Saaty stated that the relative measurement of alternatives in the AHP implied by definition that all included alternatives were relevant, in contrast to utility theory that could face rank reversal problems [ 115 ]. Apart from these fundamental questions, several authors have suggested modifications to the AHP to overcome the problem of rank reversal [ 116 ].

Our evaluation of the reported criteria emphasizes the need to increase the number of given information in AHP studies. In general, authors should improve reporting on methodology, which is essential for comprehending and reproducing other authors’ results. This would serve to facilitate other researchers’ evaluations of study quality. In our opinion, two central explanations are possible for the current underreporting in the literature. First, the AHP, being fairly new, has few precisely formulated methodological rules. Second, what rules there are do not hold in practice. The latter observation also encompasses cases in which the AHP was too difficult for participants, either because of the formulations of the criteria or because of the method itself. It can be concluded that further research, in particular, methodological research, is needed in this field.

Although this study is based on systematic literature research and transparent evaluation criteria, there are a number of limitations that bear mentioning. As we primarily conducted our research on the Pubmed and Web of Science databases, it is possible that we did not include all relevant articles from other databases, even though we conducted a manual research. In addition, not all studies reported their procedures and methodologies in detail; therefore, the resulting statements in this review and the evaluation of the studies’ reporting could be biased, as we were restricted to available information. We are unable to make statements about the appropriateness of the evaluated content, like the sample size. By contrast, our evaluation criteria considered only whether a point was mentioned. Furthermore, the evaluation of reporting relied on the CONSORT and PRISMA Statements in order to develop criteria for the AHP. These statements suggest evaluation criteria for RCTs and systematic literature reviews, thus it could be criticized that we apply them to the subjective method of the AHP. The importance of each criterion can be criticized and our overall evaluation provides only an indication of the studies’ reporting with respect to informational content—not the quality. Moreover, we summarized the articles’ procedures but were unable to convey their results without some adaptions and generalizations; some aspects of the AHP must be adapted to suit the situation.

We found that there is a pressing need to develop methodological standards for the AHP; otherwise, discrepancies in methodology could bias studies’ results. In particular, future research should establish a standard procedure for aggregating individual data, specifically, a standard for using the geometric mean versus the arithmetic mean and aggregating judgments or priorities. We should place special emphasis on finding practical sensitivity analysis to address the criticisms regarding rank reversal due to changed judgments. In addition, suggestions are necessary for reporting the robustness of weights for AHPs that do not include alternatives.

Besides the methodological aspects of the AHP, we should also think about the topic that is researched. We carved out that the AHP is based on the hierarchical structure and the criteria that are included. If the author uses improper assumptions, he will find biased results. Therefore, the AHP hierarchy should not only base on one source of information but also on a combination of different methods (e.g. literature research and expert interview). Hence, further research is required about how to determine the interviewees, what should be done with inconsistent answers, and how the outcomes and the stability of the results should be presented. In the future, we need new insights as to which target groups can best handle the challenges of the AHP. These challenges are mainly consistent answering, preventing overstraining by using adequate numbers of pairwise comparisons, and deciding between group and individual AHP. Therefore, researchers should investigate specific groups, like elderly people, healthy people, and patients with different diseases or disabilities.

In our study, we analyzed whether authors reported important aspects of the AHP in their studies. This could be a first step to evaluate the quality of studies applying AHP in healthcare. In addition, guidelines should be formulated as to which statistics should be reported and how to conduct high-quality AHPs. As mentioned before, Bridges et al. published a checklist that contains recommendations for conducting conjoint analyses on healthcare topics on behalf of the International Society for Pharmacoeconomics and Outcomes Research (ISPOR) group [ 9 ]. Besides aspects for study presentation, it suggests criteria for evaluating the choice of attributes and the appropriateness of the method for the research question. Still further, we should take the current criticisms of the AHP into consideration so that we can find solutions to address them.

This systematic literature review shows a heterogeneous picture for application of the AHP in health economics research. It is likely that interest in the AHP will rise in the future, particularly in its application to health economic evaluations, the weighing of therapy outcomes, and benefit assessments. In this context, the AHP method could support decision making regarding reimbursement of pharmaceuticals. This is largely owing to its ability to translate complex questions into stepwise comparisons at different hierarchical levels. In these hierarchies, both quantitative and qualitative criteria can be compared, which provides a more accurate representation of real-world healthcare issues. Therefore, it should be used for complex decision problems that can completely be decomposed into a hierarchical structure. Thus, patients could apply the AHP to clarify their priorities. The patients could also benefit from these structured decisions in conversations with their physicians. The second important point is to figure out by researches which are the appropriate participants that are able to judge this research problem reliably.

Abbreviations

• Analytic Hierarchy Process

Center for Health Economics Research Hannover

Consolidated Standards of Reporting Trials

Consistency Ratio

Eigenvector method

Institute for Quality and Efficiency in Health Care

International Society for Pharmacoeconomics and Outcomes Research

Preferred Reporting Items for Systematic Reviews and Meta-Analyses

Gesetz zur Neuordnung des Arzneimittelmarktes in der gesetzlichen Krankenversicherung (Arzneimittelneuordnungsgesetz –AMNOG).

Hailey D, Nordwall M. Survey on the involvement of consumers in health technology assessment programs. Int J Technol Assess Health Care. 2006;22(4):497–9.

PubMed   Google Scholar  

Buttorff C. What should be the role of patient preferences in making health care resource allocation decisions?; Available from: http://www.ispor.org/News/articles/August10/What-Should-Be-the-Role-of-Patient-Preferences.asp .

Bruera E, Sweeney C, Calder K, Palmer L, Benisch-Tolley S. Patient preferences versus physician perceptions of treatment decisions in cancer care. J Clin Oncol. 2001;19:2883–5.

PubMed   CAS   Google Scholar  

Mühlbacher AC, Juhnke C. Patient preferences versus physicians’ judgement: does it make a difference in healthcare decision making? Appl Health Econ Health Policy. 2013;11:163–80.

Article   PubMed   Google Scholar  

Gaston CM, Mitchell G. Information giving and decision-making in patients with advanced cancer: a systematic review. Soc Sci Med. 2005;61:2252–64.

Dolan JG. Multi-criteria clinical decision support: A primer on the use of multiple criteria decision making methods to promote evidence-based, patient-centered healthcare. Patient. 2010;3:229–48.

Article   PubMed   PubMed Central   Google Scholar  

Institute for Quality and Efficiency in Health Care. Allgemeine Methoden: Entwurf für Version 4.2 vom 18.06.2014. [November 27, 2014]; Available from: https://www.iqwig.de/download/IQWiG_Methoden_Entwurf-fuer-Version-4-2.pdf .

Bridges JFP, Hauber AB, Marshall D, Lloyd A, Prosser LA, Regier DA, et al. Conjoint analysis applications in health--a checklist: a report of the ISPOR Good Research Practices for Conjoint Analysis Task Force. Value Health. 2011;14:403–13.

Marshall D, Bridges JFP, Hauber B, Cameron R, Donnalley L, Fyie K, et al. Conjoint Analysis Applications in Health - How are Studies being Designed and Reported?: An Update on Current Practice in the Published Literature between 2005 and 2008. The patient. 2010;3:249–56.

Ryan M. Using conjoint analysis to elicit preferences for health care. BMJ. 2000;320:1530–3.

Article   PubMed   PubMed Central   CAS   Google Scholar  

Saaty TL. A scaling method for priorities in hierarchical structures. J Math Psychol. 1977;15:234–81.

Article   Google Scholar  

Saaty TL. The analytic hierarchy process: planning, priority setting, resource allocation 1980.

Dolan JG. Medical decision making using the analytic hierarchy process: choice of initial antimicrobial therapy for acute pyelonephritis. Med Decis Making. 1989;9:51–6.

Article   PubMed   CAS   Google Scholar  

Dolan JG, Isselhardt Jr BJ, Cappuccio JD. The analytic hierarchy process in medical decision making: a tutorial. Med Decis Making. 1989;9:40–50.

Liberatore MJ, Nydick RL. The analytic hierarchy process in medical and health care decision making: A literature review. Eur J Oper Res. 2008;189:194–207.

Cook DR, Staschak S, Green WT. Equitable allocation of livers for orthotopic transplantation: an application of the Analytic Hierarchy Process. Eur J Oper Res. 1990;48:49–56.

Dolan JG, Bordley DR, Miller H. Diagnostic strategies in the management of acute upper gastrointestinal bleeding: patient and physician preferences. J Gen Intern Med. 1993;8:525–9.

Cheever MA, Allison JP, Ferris AS, Finn OJ, Hastings BM, Hecht TT, et al. The prioritization of cancer antigens: a national cancer institute pilot project for the acceleration of translational research. Clin Cancer Res. 2009;15:5323–37.

Joshi V, Narra VR, Joshi K, Lee K, Melson D. PACS Administrators’ and Radiologists’ Perspective on the Importance of Features for PACS Selection. J Digit Imaging. 2014;27:486–95.

de Bekker-Grob EW, Ryan M, Gerard K. Discrete choice experiments in health economics: a review of the literature. Health Econ. 2012;21:145–72.

Hummel M, IJzerman M (eds.). The past and future of the AHP in health care decision making; 2011.

Mühlbacher A, Kaczynski A. Der Analytic Hierarchy Process (AHP): Eine Methode zur Entscheidungsunterstützung im Gesundheitswesen. PharmacoEcon Ger Res Artic. 2013;11:119–32.

Saaty RW. The analytic hierarchy process—what it is and how it is used. Mathematical Modelling. 1987;9:161–76.

Dolan JG, Boohaker E, Allison J, Imperiale TF. Patients’ preferences and priorities regarding colorectal cancer screening. Med Decis Making. 2013;33:59–70.

Pecchia L, Martin JL, Ragozzino A, Vanzanella C, Scognamiglio A, Mirarchi L, et al. User needs elicitation via analytic hierarchy process (AHP). A case study on a Computed Tomography (CT) scanner. BMC Med Inform Decis Mak. 2013;13:2.

Srdjevic B. Combining different prioritization methods in the analytic hierarchy process synthesis. Comput Oper Res. 2005;32:1897–919.

Saaty TL. Decision making with the analytic hierarchy process. International journal of services sciences. 2008;1:83–98.

Meixner O, Haas R. Wissensmanagement und Entscheidungstheorie: Mit 35 Tabellen. Wien: Facultas.wuv; 2010.

Forman E, Peniwati K. Aggregating individual judgments and priorities with the analytic hierarchy process. Eur J Oper Res. 1998;108:165–9.

Chen H, Kocaoglu DF. A sensitivity analysis algorithm for hierarchical decision models. Eur J Oper Res. 2008;185:266–88.

Saaty TL, Vargas LG. Sensitivity analysis in the analytic hierarchy process. In: Saaty TL, Vargas LG, editors. Decision making with the analytic network process. Boston: Springer US; 2013. p. 345–60.

Chapter   Google Scholar  

Arbel A. Approximate articulation of preference and priority derivation. Eur J Oper Res. 1989;43:317–26.

Moreno-Jimenez JM, Vargas LG. A probabilistic study of preference structures in the analytic hierarchy process with interval judgments. Math Comput Model. 1993;17:73–81.

Sugihara K, Tanaka H. Interval evaluations in the analytic hierarchy process by possibility analysis. Computational Intell. 2001;17:567–79.

Triantaphyllou E, Sánchez A. A sensitivity analysis approach for some deterministic multi-criteria decision-making methods. Decis Sci. 1997;28:151–94.

Sowlati T, Assadi P, Paradi JC. Developing a mathematical programming model for sensitivity analysis in analytic hierarchy process. IJMOR. 2010;2:290.

Masuda T. Hierarchical sensitivity analysis of priority used in analytic hierarchy process. Int J of Systems Sc. 1990;21:415–27.

Huang Y. Enhancement on sensitivity analysis of priority in analytic hierarchy process. Int J Gen Syst. 2010;31:531–42.

Erkut E, Tarimcilar M. On sensitivity analysis in the analytic hierarchy process. IMA J Management Math. 1991;3:61–83.

Altuzarra A, Moreno-Jiménez JM, Salvador M. Consensus building in AHP-group decision making: A Bayesian approach. Oper Res. 2010;58:1755–73.

Wang Y, Luo Y. On rank reversal in decision analysis. Math Comput Model. 2009;49:1221–9.

Article   CAS   Google Scholar  

Moher D, Schulz KF, Altman DG. The CONSORT statement. Revised recommendations for improving the quality of reports of parallel group randomized trails. BMC Med Res Methodol. 2001;1:2.

Moher D. Preferred reporting items for systematic reviews and meta-analyses: The PRISMA statement. Ann Intern Med. 2009;151:264.

Bi Y, Lai D, Yan H. Synthetic evaluation of the effect of health promotion: impact of a UNICEF project in 40 poor western counties of China. Public Health. 2010;124:376–91.

Karagiannidis A, Papageorgiou A, Perkoulidis G, Sanida G, Samaras P. A multi-criteria assessment of scenarios on thermal processing of infectious hospital wastes: a case study for Central Macedonia. Waste Manag. 2010;30:251–62.

Kitamura Y. Decision-making process of patients with gynecological cancer regarding their cancer treatment choices using the analytic hierarchy process. Jpn J Nurs Sci. 2010;7:148–57.

Smith J, Cook A, Packer C. Evaluation criteria to assess the value of identification sources for horizon scanning. Int J Technol Assess Health Care. 2010;26:348–53.

Taghipour H, Mohammadyarei T, Asghari Jafarabadi M, Asl HA. On-site or off-site treatment of medical waste: a challenge. J Environ Health Sci Eng. 2014;12:68.

Cabrera-Barona P, Murphy T, Kienberger S, Blaschke T. A multi-criteria spatial deprivation index to support health inequality analyses. Int J Health Geogr. 2015;14:11.

Cancela J, Fico G, Arredondo Waldmeyer MT. Using the Analytic Hierarchy Process (AHP) to understand the most important factors to design and evaluate a telehealth system for Parkinson’s disease. BMC Med Inform Decis Mak. 2015;15 Suppl 3:S7.

Lee WC, Hung FH, Tsang KF, Tung HC, Lau WH, Rakocevic V, et al. A speedy cardiovascular diseases classifier using multiple criteria decision analysis. Sensors (Basel). 2015;15:1312–20.

Lu L, Cheng H, Liu X, Xie J, Li Q, Zhou T. Assessment of regional human health risks from lead contamination in Yunnan province, southwestern China. PLoS One. 2015;10:e0119562.

Moslehi S, Atefi Manesh P, Sarabi AA. Quality measurement indicators for Iranian Health Centers. Med J Islam Repub Iran. 2015;29:177.

PubMed   PubMed Central   Google Scholar  

Mühlbacher AC, Bethge S, Kaczynski A, Juhnke C. Objective Criteria in the Medicinal Therapy for Type II Diabetes: An Analysis of the Patients’ Perspective with Analytic Hierarchy Process and Best-Worst Scaling. Gesundheitswesen. 2015. https://www.thieme-connect.com/DOI/DOI?10.1055/s-0034-1390474 .

Papadopoulos A, Sioen I, Cubadda F, Ozer H, Basegmez HIO, Turrini A, et al. TDS exposure project: application of the analytic hierarchy process for the prioritization of substances to be analyzed in a total diet study. Food Chem Toxicol. 2015;76:46–53.

Ramezanpour B, Pronker ES, Kreijtz JHCM, Osterhaus ADME, Claassen E. Market implementation of the MVA platform for pre-pandemic and pandemic influenza vaccines: A quantitative key opinion leader analysis. Vaccine. 2015;33:4349–58.

Xu X, Cao Y, Luan X. Application of 4G wireless network-based system for remote diagnosis and nursing of stomal complications. Int J Clin Exp Med. 2014;7:4554–61.

Xu Y, Levy BT, Daly JM, Bergus GR, Dunkelberg JC. Comparison of patient preferences for fecal immunochemical test or colonoscopy using the analytic hierarchy process. BMC Health Serv Res. 2015;15:175.

Mühlbacher AC, Juhnke C, Kaczynski A. Patients’ Priorities in the Treatment of Neuroendocrine Tumours: An Analytical Hierarchy Process. Gesundheitswesen. 2015. https://www.thieme-connect.com/DOI/DOI?10.1055/s-0035-1548932 .

Dou L, Yin A, Hao M, Lu J. An evaluation system for financial compensation in traditional Chinese medicine services. Complement Ther Med. 2015;23:637–43.

Zhu Q, Liu T, Lin H, Xiao J, Luo Y, Zeng W, et al. The spatial distribution of health vulnerability to heat waves in Guangdong Province. China Glob Health Action. 2014;7:25051.

Hsu JC, Tang DH, Lu CY. Risk-benefit assessment of oral phosphodiesterase type 5 inhibitors for treatment of erectile dysfunction: a multiple criteria decision analysis. Int J Clin Pract. 2015;69:436–43.

Kadohira M, Hill G, Yoshizaki R, Ota S, Yoshikawa Y. Stakeholder prioritization of zoonoses in Japan with analytic hierarchy process method. Epidemiol Infect. 2015;143:1477–85.

Hsu JC, Hsieh C, Yang YK, Lu CY. Net clinical benefit of oral anticoagulants: a multiple criteria decision analysis. PLoS One. 2015;10:e0124806.

Jaberidoost M, Olfat L, Hosseini A, Kebriaeezadeh A, Abdollahi M, Alaeddini M, et al. Pharmaceutical supply chain risk assessment in Iran using analytic hierarchy process (AHP) and simple additive weighting (SAW) methods. J Pharm Policy Pract. 2015;8:9.

Hou D, Ge X, Huang P, Zhang G, Loaiciga H. A real-time, dynamic early-warning model based on uncertainty analysis and risk assessment for sudden water pollution accidents. Environ Sci Pollut Res Int. 2014;21:8878–92.

Hu H, Liang W, Liu M, Li L, Li Z, Li T, et al. Establishment and evaluation of a model of a community health service in an underdeveloped area of China. Public Health. 2010;124:206–17.

Basoglu N, Daim TU, Topacan U. Determining patient preferences for remote monitoring. J Med Syst. 2012;36:1389–401.

Chen L, Chan C, Lee H, Chung Y, Lai F. Development of a decision support engine to assist patients with hospital selection. J Med Syst. 2014;38:59.

Chung K, Chen L, Chang Y, Chang Y, Lai M. Application of the analytic hierarchy process in the performance measurement of colorectal cancer care for the design of a pay-for-performance program in Taiwan. Int J Qual Health Care. 2013;25:81–91.

Danner M, Hummel JM, Volz F, van Manen JG, Wiegard B, Dintsios C, et al. Integrating patients’ views into health technology assessment: Analytic hierarchy process (AHP) as a method to elicit patient preferences. Int J Technol Assess Health Care. 2011;27:369–75.

Diaz-Ledezma C, Parvizi J. Surgical approaches for cam femoroacetabular impingement: the use of multicriteria decision analysis. Clin Orthop Relat Res. 2013;471:2509–16.

Joshi V, Lee K, Melson D, Narra VR. Empirical investigation of radiologists’ priorities for PACS selection: an analytical hierarchy process approach. J Digit Imaging. 2011;24:700–8.

Lambooij MS, Hummel MJ. Differentiating innovation priorities among stakeholder in hospital care. BMC Med Inform Decis Mak. 2013;13:91.

Lee CW, Kwak NK. Strategic enterprise resource planning in a health-care system using a multicriteria decision-making model. J Med Syst. 2011;35:265–75.

Li A, Lin J. Constructing core competency indicators for clinical teachers in Taiwan: a qualitative analysis and an analytic hierarchy process. BMC Med Educ. 2014;14:75.

Li C, Yu C. Performance evaluation of public non-profit hospitals using a BP artificial neural network: the case of Hubei Province in China. Int J Environ Res Public Health. 2013;10:3619–33.

Olivieri A, Marchetti M, Lemoli R, Tarella C, Iacone A, Lanza F, et al. Proposed definition of ‘poor mobilizer’ in lymphoma and multiple myeloma: an analytic hierarchy process by ad hoc working group Gruppo ItalianoTrapianto di Midollo Osseo. Bone Marrow Transplant. 2012;47:342–51.

Page K. The four principles: can they be measured and do they predict ethical decision making? BMC Med Ethics. 2012;13:10.

Pecchia L, Bath PA, Pendleton N, Bracale M. Analytic Hierarchy Process (AHP) for examining healthcare professionals’ assessments of risk factors. The relative importance of risk factors for falls in community-dwelling older people. Methods Inf Med. 2011;50:435–44.

Sharma PS, Eden KB, Guise J, Jimison HB, Dolan JG. Subjective risk vs. objective risk can lead to different post-cesarean birth decisions based on multiattribute modeling. J Clin Epidemiol. 2011;64:67–78.

Suner A, Celikoglu CC, Dicle O, Sokmen S. Sequential decision tree using the analytic hierarchy process for decision support in rectal cancer. Artif Intell Med. 2012;56:59–68.

Bahadori M, Ravangard R, Yaghoubi M, Alimohammadzadeh K. Assessing the service quality of Iran military hospitals: Joint Commission International standards and Analytic Hierarchy Process (AHP) technique. J Educ Health Promot. 2014;3:98.

Mok H, Zhou Y, Chen J, Gao Q. Development and validation of a convenient formula evaluating the value and applicability of medical literature in clinical practice. Pak J Med Sci. 2014;30:1377–82.

Reddy BP, Kelly MP, Thokala P, Walters SJ, Duenas A. Prioritising public health guidance topics in the National Institute for Health and Care Excellence using the Analytic Hierarchy Process. Public Health. 2014;128:896–903.

Shojaei P, Karimlou M, Nouri J, Mohammadi F, Malek Afzali H, Forouzan AS. Ranking the effects of urban development projects on social determinants of health: health impact assessment. Glob J Health Sci. 2014;6:183–95.

Šoltés V, Gavurová B. The functionality comparison of the health care systems by the analytical hierarchy process method. E + M 2014;17:100–17.

Tu C, Fang Y, Huang Z, Tan R. Application of the analytic hierarchy process to a risk assessment of emerging infectious diseases in Shaoxing city in southern China. Jpn J Infect Dis. 2014;67:417–22.

Hsu H, Tsai C, Chang M, Luh D. Constructing area-level indicators of successful ageing in Taiwan. Health Soc Care Community. 2010;18:70–81.

Lin R, Chuang C. A hybrid diagnosis model for determining the types of the liver disease. Comput Biol Med. 2010;40:665–70.

Ajami S, Ketabi S. Performance evaluation of medical records departments by analytical hierarchy process (AHP) approach in the selected hospitals in Isfahan: Medical records dep. & AHP. J Med Syst. 2012;36:1165–71.

Hilgerink MP, Hummel MJ, Manohar S, Vaartjes SR, Ijzerman MJ. Assessment of the added value of the Twente Photoacoustic Mammoscope in breast cancer diagnosis. Med Devices (Auckl). 2011;4:107–15.

Hummel JM, Boomkamp ISM, Steuten LMG, Verkerke BGJ, Ijzerman MJ. Predicting the health economic performance of new non-fusion surgery in adolescent idiopathic scoliosis. J Orthop Res. 2012;30:1453–8.

Ijzerman MJ, van Til JA, Bridges JFP. A comparison of analytic hierarchy process and conjoint analysis methods in assessing treatment alternatives for stroke rehabilitation. Patient. 2012;5:45–56.

Perseghin P, Marchetti M, Pierelli L, Olivieri A, Introna M, Lombardini L, et al. A policy for the disposal of autologous hematopoietic progenitor cells: report from an Italian consensus panel. Transfusion. 2014;54:2353–60.

Uzoka FE, Obot O, Barker K, Osuji J. An experimental comparison of fuzzy logic and analytic hierarchy process for medical decision support systems. Comput Methods Programs Biomed. 2011;103:10–27.

Kuruoglu E, Guldal D, Mevsim V, Gunvar T. Which family physician should I choose? The analytic hierarchy process approach for ranking of criteria in the selection of a family physician. BMC Med Inform Decis Mak. 2015;15:63.

Riepe MW. Clinical preference for factors in treatment of geriatric depression. Neuropsychiatr Dis Treat. 2015;11:25–31.

Krishnamoorthy K, Mahalingam M. Selection of a suitable method for the preparation of polymeric nanoparticles: multi-criteria decision making approach. Adv Pharm Bull. 2015;5:57–67.

Kunasekaran V, Krishnamoorthy K. Multi criteria decision making to select the best method for the preparation of solid lipid nanoparticles of rasagiline mesylate using analytic hierarchy process. J Adv Pharm Technol Res. 2014;5:115–21.

Velmurugan R, Selvamuthukumar S, Manavalan R. Multi criteria decision making to select the suitable method for the preparation of nanoparticles using an analytical hierarchy process. Pharmazie. 2011;66:836–42.

Wollmann D, Steiner MT, Vieira GE, Steiner PA. Evaluation of health service providers by consumers through the Analytic Hierarchy Process Method. Rev Saude Publica. 2012;46:777–83.

Fang L, Tung H. Comparison of nurse practitioner job core competency expectations of nurse managers, nurse practitioners, and physicians in Taiwan. J Am Acad Nurse Pract. 2010;22:409–16.

Maruthur NM, Joy S, Dolan J, Segal JB, Shihab HM, Singh S. Systematic assessment of benefits and risks: study protocol for a multi-criteria decision analysis using the Analytic Hierarchy Process for comparative effectiveness research. F1000Res. 2013;2:160.

Zhang S, Wei Z, Liu W, Yao L, Suo W, Xing J, et al. Indicators for Environment Health Risk Assessment in the Jiangsu Province of China. Int J Environ Res Public Health. 2015;12:11012–24.

Diaz-Ledezma C, Lichstein PM, Dolan JG, Parvizi J. Diagnosis of periprosthetic joint infection in medicare patients: Multicriteria decision analysis. Clin Orthop Relat Res. 2014;472(11):3275–84.

Petit J, Meurice N, Kaiser C, Maggiora G. Softening the rule of five. Where to draw the line? Bioorg Med Chem. 2012;20:5343–51.

Munoz DA, Nembhard HB, Kraschnewski JL. Quantifying complexity in translational research: an integrated approach. Int J Health Care Qual Assur. 2014;27:760–76.

Guariguata L, Whiting D, Weil C, Unwin N. The International Diabetes Federation diabetes atlas methodology for estimating global and national prevalence of diabetes in adults. Diabetes Res Clin Pract. 2011;94:322–32.

Maruthur NM, Joy SM, Dolan JG, Shihab HM, Singh S. Use of the analytic hierarchy process for medication decision-making in type 2 diabetes. PLoS One. 2015;10:e0126625.

Dyer RF, Forman EH. Group decision support with the Analytic Hierarchy Process. Decis Support Syst. 1992;8:99–124.

Saaty TL, Shang JS. Group decision-making: Head-count versus intensity of preference. Socio Econ Plan Sci. 2007;41:22–37.

Broekhuizen H, Groothuis-Oudshoorn CGM, van Til JA, Hummel JM, Ijzerman MJ. A review and classification of approaches for dealing with uncertainty in multi-criteria decision analysis for healthcare decisions. Pharmacoeconomics. 2015;33:445–55.

Saaty TL. How to make a decision: The analytic hierarchy process. Eur J Oper Res. 1990;48:9–26.

Maleki H, Zahir S. A comprehensive literature review of the rank reversal phenomenon in the analytic hierarchy process. J Multi-Crit Decis Anal. 2013;20:141–55.

Curran SS, Tkach VV, Overstreet RM. A new species of Homalometron (Digenea: Apocreadiidae) from fishes in the northern Gulf of Mexico. J Parasitol. 2013;99:93–101.

Download references

Acknowledgements

The Center for Health Economics Research Hannover (CHERH) is founded by the Federal Ministry of Education and Research.

Author information

Authors and affiliations.

Center for Health Economics Research Hannover (CHERH), Leibniz University of Hanover, Otto-Brenner-Str. 1, 30159, Hannover, Germany

Katharina Schmidt, Ines Aumann, Kathrin Damm & J.-Matthias Graf von der Schulenburg

Institute for Risk and Insurance, Leibniz University of Hanover, Otto-Brenner-Str. 1, 30159, Hannover, Germany

Ines Hollander

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Katharina Schmidt .

Additional information

Competing interests.

The author(s) declare that they have no competing interests.

Authors’ contributions

KS carried out the analyses and drafted the manuscript. IA and IH participated in the review process and decision making process for identifying relevant articles. IA made substantial contributions to conception of the article. IH collected and prepared the data adequately for the manuscript. KD participated in selection process of papers and she revised the manuscript. JMS revised the manuscript for important intellectual content. All authors read and approved the final manuscript.

Rights and permissions

Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License ( http://creativecommons.org/licenses/by/4.0/ ), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated.

Reprints and permissions

About this article

Cite this article.

Schmidt, K., Aumann, I., Hollander, I. et al. Applying the Analytic Hierarchy Process in healthcare research: A systematic literature review and evaluation of reporting. BMC Med Inform Decis Mak 15 , 112 (2015). https://doi.org/10.1186/s12911-015-0234-7

Download citation

Received : 14 August 2015

Accepted : 15 December 2015

Published : 24 December 2015

DOI : https://doi.org/10.1186/s12911-015-0234-7

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Multi-criteria decision making
• Methodological standards

BMC Medical Informatics and Decision Making

ISSN: 1472-6947

• General enquiries: [email protected]

Click through the PLOS taxonomy to find articles in your field.

For more information about PLOS Subject Areas, click here .

Loading metrics

Open Access

Peer-reviewed

Research Article

State-of-the-art on analytic hierarchy process in the last 40 years: Literature review based on Latent Dirichlet Allocation topic modelling

Roles Conceptualization, Funding acquisition, Methodology, Project administration, Visualization, Writing – original draft, Writing – review & editing

* E-mail: [email protected]

Affiliation Department of Business Administration and Management, Technical University of Liberec, Liberec, Czech Republic

Roles Conceptualization, Data curation, Investigation, Methodology, Resources, Software, Validation, Writing – original draft, Writing – review & editing

Affiliation Department of Macro and Microeconomy, University of Žilina, Žilina, Slovakia

• Peter Madzík, 
• Lukáš Falát

• Published: May 27, 2022
• https://doi.org/10.1371/journal.pone.0268777
• Reader Comments

Although there are several articles that have carried out a systematic literature review of the analytical hierarchy process (AHP), many of them work with a limited number of analyzed documents. This article presents a computer-aided systematic literature review of articles related to AHP. The objectives are: (i) to identify AHP usage and research impact in different subject areas; (ii) to identify trends in the popularity of the AHP from the first introduction of the method in 1980 to the present; (iii) to identify the most common topics related to AHP and topic development over time. We process 35,430 documents related to AHP, published between 1980 and 2021, retrieved from the Scopus database. We provide detailed statistics about research interest, research impact in particular subject areas over the analyzed time period. We use Latent Dirichlet Allocation (LDA) using Gibbs sampling to perform topic modeling based on the corpus of abstracts. We identify nine topics related to AHP: Ecology & Ecosystems; Multi-criteria decision-making; Production and performance management; Sustainable development; Computer network, optimization and algorithms; Service quality; Fuzzy logic; Systematic evaluation; Risk assessment. We also present the individual topics trends over time and point out the possible future direction of AHP.

Citation: Madzík P, Falát L (2022) State-of-the-art on analytic hierarchy process in the last 40 years: Literature review based on Latent Dirichlet Allocation topic modelling. PLoS ONE 17(5): e0268777. https://doi.org/10.1371/journal.pone.0268777

Editor: Yiming Tang, Hefei University of Technology, CHINA

Received: January 11, 2022; Accepted: May 7, 2022; Published: May 27, 2022

Copyright: © 2022 Madzík, Falát. This is an open access article distributed under the terms of the Creative Commons Attribution License , which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.

Data Availability: The data is available at following URL: https://figshare.com/articles/dataset/AHP_data_xlsx/18176240 .

Funding: This publication has been supported by the grant CZ.02.2.69/0.0/0.0/18_053/0017628 International Mobilities at the TUL II. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.

Competing interests: The authors have declared that no competing interests exist.

1 Introduction

The Analytic Hierarchy Process (AHP) method is currently one of the most frequently used decision support tools. Saaty [ 1 ] proposed the initial logic of the method, and three years later, the AHP method was generalized as a universal decision support tool. AHP is based on three principles that one can recognize in problem-solving—decomposition, comparative judgments, and synthesis of priorities [ 2 ]. The procedure proposed by Saaty significantly simplifies prioritization in multiple criteria decision-making. To date, many research papers have been published that have used this method and have focused on areas such as selection, evaluation, benefit-cost, allocation, planning and development, priority and ranking, decision making (general), forecasting, medicine, or quality function deployment [ 3 ]. The application of AHP is also not limited by industry and finds application in virtually any research area [ 4 ].

If we look at the use of the AHP method in research and practical applications, we can state that the usage of the AHP is very widespread. The most common articles to cover state-of-the-art in any field are systematic reviews. Systematic review studies or meta-analysis studies focused on AHP have in the past sought to capture the main currents and directions in the use of this method. Although many presented AHP applications and identified mainstream streams, most of them could have two main limitations. The first one could be questionable representativeness. Systematic literature review articles are most often analyzed articles in the most reputable journals. Although they captured the strongest trends, analysis of this type rarely covered more than 100 such articles. At the same time, it should be noted that the primary objective of systematic review is synthesizing evidence [ 5 , 6 ]. Systematic reviews are often focused on answering specific issues in a specific field. General research questions are rarely the subject of systematic reviews. On the other hand, it is understandable that a systematic literature review, which would contain thousands of articles, would be extremely time-consuming in terms of implementation. The second limitation is the timeliness of the findings found in the systematic literature review or meta-analysis review articles. The longer it has been since the study was published, the less current its conclusions are. In connection with the intensive growth of the use of AHP, the need for up-to-date trend capture is becoming increasingly important. This article focuses on the data-driven comprehensive review of AHP use, analyzing huge amount of Scopus documents. Compared to standard systematic reviews, this study could provide a broader picture of AHP method.

1.1 Foundations of analytic hierarchy process

AHP has undergone dynamic development since its inception, but in the 1980s, researchers focused more on developing the principles and foundations of this method. At a certain degree of simplification, it can be stated today that the method has three basic principles and three axioms [ 7 , 8 ]. The first principle is comparative judgments to determine the "local" priorities (weight) of the elements. The other two principles—the principle of hierarchical composition and the principle of synthesis—make it possible to process local priorities into "global" priorities. To apply these principles, researchers often refer to three axioms. The first is the reciprocal axiom, which requires a pairwise comparison of elements. The second is the homogeneity axiom. It should not be used to compare widely disparate elements [ 9 ]. The third is the synthesis of axioms that states that judgments about or the priorities of the elements in a hierarchy do not depend on lower level elements [ 9 ]. While the first two axioms are generally fully sufficient for practical purposes, according to Forman and Gass [ 8 ], the third axiom should evoke discourse.

One of the main advantages of AHP is its flexibility, logic, and ease of application, which has been reflected in the significant growth of publications that use this method [ 4 ]. Decision-making can be found in virtually any research area—for this reason, the application of AHP has been applied in areas such as engineering [ 10 ], computer science [ 11 ], business and management [ 12 ], mathematics [ 13 ] or social sciences [ 14 ]. The possibilities of AHP adjustments are also relatively wide, for example, through fuzzy logic [ 15 ], sensitivity analysis [ 16 ], or application to problems associated with risk assessment [ 17 ], or design [ 18 ]. However, these AHP applications represent only a selection of the most common and more detailed information on the possibilities of AHP is provided by systematic literature review papers on this method.

1.2 State of the art of AHP reviews

The development of the AHP method and its application had a relatively wide application in the 1980s. However, these applications relied heavily on developing the mathematical foundations of AHP [ 19 ]. The first review article on the possibilities of applying AHP was published by Vargas [ 20 ]. In his work, he summarized the methodological foundations of the use of AHP and its axioms and synthesized research articles published so far. The results pointed out that AHP can be used to solve economic/managerial, political, social, or technological problems [ 20 ].

The growing interest in AHP applications is documented by a brief look at the Scopus bibliographic and citation database. Between 1980 and 1990, 109 articles related to AHP were registered in this database. In the next period 1991–2000, the increase in such records was almost 6-fold (a total of 604 articles). This increase is mainly characterized by the extension of the AHP method to other scientific areas. Concerning the impact of the articles measured over the number of citations, some of the most important articles can be described in more detail.

Forman and Gass [ 8 ] published a study, the aim of which was to discuss why AHP is a general methodology for a wide variety of decisions and other applications, to present brief descriptions of successful applications of the AHP and to elaborate on academic discourses relevant to the efficacy and applicability of the AHP vis-a-vis competing methodologies. Based on the analysis of the successful use of AHP in various companies and institutions, the authors defined eight application areas: choice, prioritization/evaluation, resource allocation, benchmarking, quality management, public policy, health care, strategic planning. This practical part was extended by a scientific discourse focused on six areas: transitivity and rank reversal, transitivity, adding irrelevant alternatives and rank reversal, measurement and ratio-scales, prioritizing objectives/criteria, AHP with feedback (ANP) and approximations. The study’s strength is a relatively detailed overview of the principles and foundations of AHP and an attempt to define the application areas.

Another review study was published by Vaidya and Kumar [ 3 ]. It aimed to present a literature review of AHP applications. The authors analyzed 150 selected articles related to AHP, which were published before 2003. The articles were subsequently analyzed according to three aspects: applications based on a theme; specific applications; applications combined with some other methodology. The presented results were divided into ten application areas: selection; evaluation; benefit-cost analysis; allocations; planning and development; priority and ranking; decision-making; forecasting; medicine and related fields; AHP in QFD applications. According to this categorization, it can be seen that the views on the classification are partially mixed and include a purpose perspective (selection, evaluation, allocation, etc.) as well as a sectoral or sectoral perspective (medicine, QFD). The study also contains an overview of the most frequently used journals for publishing topics related to AHP, which is positive. The authors also tried to outline the development of the topic of AHP over time but used only a simple overview in the form of a pie chart, which covers periods of three to four years. On the other hand, the authors should be commended for the content analysis of a large number of articles, without which the definition of thematic groups would not be possible.

In 2010, the Turkish authors Sipahi and Timor [ 21 ] published another review study on the current possibilities of using AHP and its extended version of ANP. This literature review included an analysis of 232 application articles related to AHP or ANP in the period 2005–2009. Based on the content analysis of selected articles, the authors found that an exponential increase in the application of AHP can be observed in the observed period. The article offers a relatively good overview of the original sources in each area, and the structure is somewhat reminiscent of the study by Forman and Gass [ 8 ]. The authors supplement the results with the combination possibilities of AHP, as they also give examples when this method is used together with other tools such as simulation, TOPSIS, GIS, Goal programming, etc. The positive aspect of this study can be considered the relatively high number of analyzed articles. On the other hand, the negative of this literature review can be considered a narrow period of time, which can offer the current state of the AHP application, but without the possibility to capture the past development of this topic.

While previous literature analyzes have focused on defining the application areas of AHP, the study by Ishizaka and Labib [ 16 ] focused more on methodological developments of AHP. The study aimed to conduct a neutral review of nine methodological topics that the researchers had addressed in the past. These topics include problem modeling, pairwise comparison, judgment scales, derivation methods, consistency indices, incomplete matrix, synthesis of the weights, sensitivity analysis, and group decisions. Although the authors deal with these topics mathematically, they also state that the success of the use of AHP is its simplicity, hierarchical modeling of the problem, and the possibility of adopting verbal judgments. This review study offered a relatively new and original overview of the use of AHP not through a purposeful and sectoral perspective but through methodological issues.

One of the broadest review studies on the AHP applications includes a literature review with a social networks analysis, published by Emrouznejad and Marra [ 4 ]. This study aimed to trace the pattern of development of AHP research, identify the patterns of collaboration among authors, identify the most important papers underpinning the development of AHP and discover recent areas of interest. Regarding the number of articles examined, this study is the most extensive of all mentioned—8441 papers published between 1979 and 2017 retrieved from the ISI Web of Science database were analyzed. The results, to some extent, confirmed previously published findings regarding the development of the AHP topic. The authors identified in the first time period (1979–1990) that attention was focused on the development of the theoretical foundations of AHP. In the second period (1991–2001), there was an increase in the application of AHP in areas such as computer science, mathematics, business, and management studies and its introduction in new research areas. The third period, which covered the years 2002–2017, was characterized by expanding AHP into areas such as fuzzy logic, TOPSIS, DEAHP, SWOT, QFD, sensitivity analysis.

Five studies were presented above, which focused on a systematic analysis of the development of topics related to AHP. These studies were generically focused on the comprehensive capture of AHP without deeper and more detailed specialization. For the sake of completeness, however, it should be noted that the topic of AHP and its applications was also analyzed from a more detailed perspective, either from the perspective of a specific subject area or other characteristics. Below is a selection of some overview articles with a more specific focus:

• Apostolou and Hassell [ 22 ] summarized the use of AHP in accounting research through a chronological arrangement
• Ho [ 13 ] focused on the analysis of articles in which AHP is combined with other tools such as QFD, DEA, or SWOT
• The classification of healthcare articles according to several classification criteria (publication year, journal, method of analyzing alternatives, etc.) was published in their study by Liberatore and Nydick [ 23 ]
• Subramanian and Ramanathan [ 10 ] analyzed the development of articles in operations management and pointed to the trend of using AHP when problems require considerations of both quantitative and qualitative factors.

1.3 Research gap

From the review studies described above, one can see an effort to cover the topic of AHP as widely as possible. As the systematic literature review studies that analyze the AHP application usually included only a few dozen studies, the representativeness of the results may not always be guaranteed. Most authors of such review studies seek to address this shortcoming by including studies in the most reputable journals in the analysis. As these are highly renowned journals, this may partially reduce the representativeness problem, but it will not completely eliminate it. One of the few studies that have eliminated such a deficiency is a review conducted by Emrouznejad and Marra [ 4 ]. However, the authors of this study apparently had to proceed with simplification for interpretation reasons and divided the results into three groups according to the time period. Although the results are more complex, it was difficult to capture trends in the development of AHP.

However, bibliographic and citation databases currently offer much broader analytical possibilities for processing scientific trends in various topics or areas. Given the enormous growth of articles published on the topic of AHP over the last five years, the need to capture the trends in the application of AHP with regard to its past development is extremely topical. Our study reflects the need for a review of AHP—we use a big-data approach to go beyond the scope of systematic reviews. A data-driven machine learning approach was used to get a broader picture of AHP usage. In this article, we focus on three areas (research questions) that have so far been insufficiently taken into account in the comprehensive analysis of AHP:

• RQ1: What is the usage of AHP and research impact in individual subject areas?
• RQ2: What are the trends in AHP popularity from the first introduction of the method in 1980 to the present?
• RQ3: What are the most common topics related to AHP, and what is their development over time?

Focusing on these three research questions will make it possible to update previous results and broaden the context of AHP applications by examining a many more articles. Therefore, it can be assumed that the results will show a higher degree of representativeness than the review articles published so far.

2 Methodology

To cover the defined research questions, a procedure consisting of three main phases was determined—data acquisition, variables (dataset structure), and data analysis. These three phases are described in more detail in Chapters 2.1 to 2.3. Particular methodology steps are shown in Fig 1 .

• PPT PowerPoint slide
• PNG larger image
• TIFF original image

https://doi.org/10.1371/journal.pone.0268777.g001

2.1 Data acquisition

The acquisition process data consisted of two steps, with the data downloaded from the Scopus database. Scopus is one of the most prestigious and largest scientific databases and contains information on abstracts and citations and other metadata on scientific articles. Currently, this database includes more than 76 million records. Indexing sources come from more than 39,100 journals, 120,000 conferences, and 206,000 books [ 24 ].

As a first step, we focused our search on documents of scientific articles published between 1980 and 2021. The data presented in this article were collected on October 12, 2021. Data retrieval strategy in the Scopus database was as follows. After setting up a document search, we set the search criteria to Article title, Abstract, Keywords. We then defined a search query: ahp or "analytic hierarchy process". The returned results were further modified by removing documents that were published before 1980 (the year of the first publication of the AHP). We have obtained 35,453 documents. Finally, documents that had the release year of 2022 were removed. The resulting dataset was 35,430 documents in size. Our selection process was not limited due to the type of studies, i. e. the suitability of the studies for our sample was not limited to systematic reviews or meta-analyses.

In the second step, we obtained a database of resources indexed in Scopus. The data in this database contained the name of the source and its assignment to one or more of the 26 subject areas.

2.2 Variables

The final dataset was created by merging the two datasets described above. Data on subject areas have been paired to the document records dataset. The dataset contained 35,430 rows and 32 columns. The rows represent the documents, and the columns attributes of the individual documents. Attributes (variables) defined the basic information about the article, i. e.: authors, title, year, source title (journal name), the number of citations, the text of the abstract, and 26 subject areas, which were defined as follows: Agricultural and Biological Sciences (AGRI); Arts and Humanities (ARTS); Biochemistry, Genetics and Molecular Biology (BIOC); Business, Management and Accounting (BUSI); Chemical Engineering (CENG); Chemistry(CHEM); Computer Science (COMP); Decision Sciences (DECI); Dentistry (DENT); Earth and Planetary Sciences (EART); Economics, Econometrics and Finance (ECON); Energy (ENER); Engineering (ENGI); Environmental Science (ENVI); Health Professions (HEAL); Immunology and Microbiology (IMMU); Materials Science (MATE); Mathematics (MATH); Medicine (MEDI); Neuroscience (NEUR); Nursing (NURS); Pharmacology, Toxicology and Pharmaceutics (PHAR); Physics and Astronomy (PHYS); Psychology (PSYC); Social Sciences (SOCI); Veterinary (VETE). If the publication was included in the selected subject area by the Scopus database, this was marked for the given document. It is important to recall that one article could be included in more than one subject area.

2.3 Data analysis

Data analysis was performed in two phases. The first phase involved descriptive and exploratory data analysis. The main summary measures were the following three main metrics: number of articles, number of citations of an article, average number of citations per article. In addition, we used the metrics number of new articles (2017–2021) and the h-index [ 25 ].

In addition to the main summary measures, we used cumulative numbers and a Pareto diagram to synthesize the results of the descriptive analysis. Next, we structured the results of the descriptive analysis according to subject areas and individual years. When structuring by subject areas, we monitored the number of articles in the subject area, the average number of citations per article in the subject area, the number of new articles (2017–2021) in the subject area, the Hirsch index in the subject area. We also used relative statistics in the structured analysis according to subject areas (cumulative percentage of articles by subject area, share of articles in selected subject area over total articles, share of citations in selected subject area over total citations). When structuring by individual years, we monitored the development of the total number of articles published in selected subject areas for individual years. Finally, we analyzed the journals with the highest impact on AHP dissemination based on the total number of article citations.

The second phase of data analysis was topic modeling using the Latent Dirichlet Allocation (LDA) method. It is an unsupervised machine learning method of probabilistic clustering and is a type of Bayesian model. The principle of the method is that each element of the dtm (document-term matrix) matrix is a mixture of a finite number of topics with a certain probability. Each topic is a mixture of several words with a certain division. [ 26 ]

The practical implementation of extracting topics from data was performed in five steps: corpus creation, preprocessing, creation of dtm matrix, modeling of topics, and visualization of topics. All the above procedures were implemented in R language. For corpus creation and basic data preprocessing we used the tm package, which is a textmining package [ 27 ]. We used the SnowballC package to implement stemming, and the topicmodels package to model the topics themselves. We used LDAvis, servr, dplyr, strings, magrittr packages for visualization.

The first step that preceded the modeling of topics was the creation of a corpus, which is the set of all documents (abstracts). In our case, the corpus contained 35,430 documents and consisted of a set of all abstracts that were the input for our analysis.

The second step was to pre-process the data in the corpus, as the text data is unstructured data and, in essence, contains several problems for computer processing. In the preprocessing phase, all words in the whole corpus were transformed into lowercase, special characters (-,:, ‘,” -”, ©) were removed, punctuation was removed, numbers were removed, and additional spaces were removed. Subsequently, stemming was performed, in which the words were truncated to the word base. Subsequently, the differences between American and Australian English were removed, and non-meaningful words (stopwords) were removed from the corpus. In addition to the standard stopwords from the tm package, we also defined our own stopwords that have been removed from the corpus. Furthermore, we removed a set of so-called ahp stopwords because these words did not explain the topics, as they were found in every article.

The third step was to create a document-term matrix (dtm), whose rows contained documents (abstracts), and the columns formed words from the corpus. For computational efficiency, we decided to limit in dtm the maximum document frequency of a word to the number of documents (35430) and the minimum document frequency of a word to 1 percent (i.e. 35) of documents. We’ve also limited the minimum word length to 4 and the maximum word length to 20 characters.

Since Gibbs sampling starts at a random point, we decided to burn the first 100 steps of this process (these results did not well represent the properties of our probability distribution). Subsequently, we performed 2000 iterations of this procedure, and due to the correlation between the samples, we took only every 40th iteration for further use. We performed experiments with a number of topics from 8 to 12. In order to minimize the chance of getting stuck in the local minimum, we performed 5 runs for each value k, and we saved only the best result. For the replicability of the results, we defined the initial settings (seed) of the 5 run runs: 2003, 5, 63, 100001, 765. With regard to assessing the degree of cluster distinguishability based on the composition of the most frequent words in individual topics, we decided on the final number of topics.

The fifth step was to visualize the themes. Topics were visualized on intertopic 2D distance maps via multidimensional scaling using principal component analysis (PCA) via the LDAvis library. In the intertopic map, each topic was represented by top-30 most salient terms, where saliency was defined according to Chuang, Manning and Heer [ 31 ].

The final product of the quantified LDA method was a list of the abstracts of individual abstracts to the topic with the highest probability, a list of the most representative words to the given topic and a list of probabilities of the affiliation of each document to each topic.

3.1 AHP research in subject areas and citation overview

35,430 records were included in the analysis, which contained the terms AHP or "Analytic hierarchy process". The number of records was current as of October 12, 2021, and these articles covered the period from 1980 to 2021. A total of 457,815 citations were registered for all these articles. Fig 2 shows the Pareto article distribution report (only the 10,000 most cited records were displayed). The first 5,784 most cited articles (16.3%) had a total of 80% of all citations (366,260 citations).

https://doi.org/10.1371/journal.pone.0268777.g002

Topics related to AHP have been included in articles in virtually all subject areas. The number of these areas was 26. Fig 3 provides an overview of the number of articles, their citations, and the Hirsch index (Hirsch 2005) for each subject area. The attractiveness of the AHP can be measured by the number of citations of articles from the subject areas. However, the Citation per article indicator can be distorted, especially in cases where the total number of articles is low or there are relatively few articles with a very high number of citations. From the point of view of the attractiveness of the AHP, the Hirsch index, which combines productivity and research impact, is a better indicator. At the top of Fig 3 , we can see that AHP related themes appear most in the articles from the following subject areas: ENGI, COMP, ENVI, BUSI, SOCI, MATH, DECI and EART. The differences between the number of articles and their citations can be significant, as shown at the bottom of Fig 3 . If we wanted to define the dominant research areas related to the topic of AHP, we could select those for which the Hirsch index is higher than e.g. 100 (i.e. at least 100 articles have at least 100 citations). According to such an approach, we could include ENGI, COMP, ENVI, BUSI, MATH and DECI, among the dominant areas.

https://doi.org/10.1371/journal.pone.0268777.g003

AHP is a multiple-criteria decision-making tool, which finds application mainly in those areas in which efforts to objectify decision-making can be observed. Interestingly, however, AHP was originally most closely associated with MATH and DECI research areas. These research areas still account for a significant share of the total number of articles and the total number of citations. But we can see that AHP is a multidisciplinary topic. Research already covers areas such as ENGI, COMP, ENVI, and BUSI, which are not at all negligible in terms of academic performance and impact.

To better understand the use of AHP in various scientific fields, we can analyze the number of articles and the number of citations by the source in which they were published. Table 1 provides an overview of journals sorted by the number of citations.

https://doi.org/10.1371/journal.pone.0268777.t001

One anomaly can be observed from the table—namely, the journal Diabetes Research and Clinical Practice. Only three articles on AHP have been published there, but they have an enormous citation rate. If we exclude this extreme, the top-5 journals publishing topics on AHP include European Journal of Operational Research, Expert Systems with Applications, Journal of Cleaner Production, International Journal of Production Economics and International Journal of Production Research. These journals are among the top in their field of science, which only testifies to the relevance of AHP’s research potential.

3.2 Trends in AHP popularity

To capture the use of the AHP method in the monitored 26 subject areas, we analyzed the number of articles in the given years. Fig 4 provides an overview of the development of all subject areas. The number of articles devoted to AHP is constantly growing in virtually all areas. We will discuss only a few of the most significant findings. In the area of ENGI, a deviation from the growing trend can be observed from 2015 to 2019. According to the development, it seems that the number of articles focused on AHP in this research area is currently approximately the same as in 2013 and 2014. Strong growth of interest in AHP can be identified in the area of ENVI and SOCI. Both areas have seen a significant increase in published articles since 2015.

https://doi.org/10.1371/journal.pone.0268777.g004

The trend of publishing scientific articles has been growing for a long time. The Scopus database contains a total of 6.3 million records in 2020—articles, conference papers, reviews, editorials, notes, letters, etc.—in all subject areas. Twenty years ago—in 2000—there were only 2.1 million records and in 1980 only 0.9 million. The above results, therefore, need to be taken into account in view of this increase.

Interest in AHP topics can also be analyzed by comparing the total number of articles in a given subject area in a given year and articles focused on AHP in a given year. Such a comparison will partially eliminate interpretation problems and help identify a real interest in the topics. The results are shown in Fig 5 .

https://doi.org/10.1371/journal.pone.0268777.g005

The topic of AHP is most used in ENGI (approx. 0.50% of all articles), BUSI (approx. 0.30%), and DECI (approx. 0.30%). ENGI had the greatest research interest in AHP in 2013 and 2014 (approximately 0.80%), while it subsequently fell sharply. The steady growth of research interest can be seen in the BUSI area, growing almost continuously since about 2005. It is also interesting to note that in the area of DECI, research interest in AHP has been hovering around the level of 0.30% for almost 40 years. This recalculation has not confirmed the increase in absolute article numbers previously identified in the ENVI and SOCI areas. There are currently a lot of published articles in the areas of ENVI and SOCI, with only a fraction directly or indirectly related to AHP (approximately 0.10%).

3.3 Topics related to AHP and their development

We used LDA topic modeling to analyze topics. Topic modeling is the process of identifying topics in a set of documents. Our set of documents consisted of 35,430 articles, and the LDA was used for abstracts of these articles. Several experiments have been performed with LDA to achieve a reasonable number of clusters with good interpretability and distinguishability. Based on the settings listed in section 2.3, we identified nine topics. Different terms with different frequencies characterized each topic. Fig 6 shows the display of topics via intertopic distance maps.

https://doi.org/10.1371/journal.pone.0268777.g006

The words that were most frequent in a particular topic formed the basis for naming the topic. The higher the number of specific terms in the topic, the more we considered this term when naming the topic. Each article was assigned a probability of belonging to a given topic by the LDA algorithm. The article was assigned to the topic for which the probability was the highest. Based on this, it was possible to display the size of the topic (number of documents) and their research impact (number of citations). The identified topics were relatively independent, as the correlation coefficients between them reached low values—in the interval <-0.31; 0.06>. With regard to the most frequent words, the topics were named, and their representation in individual subject areas was assessed, while the top 4 subject areas were highlighted— Fig 7 .

https://doi.org/10.1371/journal.pone.0268777.g007

Before we describe the topics and their characteristics, we looked at the development of topics over time. We analyzed the development from two perspectives—research interest and research impact. We analyzed the research interest through the relative number of articles on each of the nine topics in each year under review. The higher number (and proportion) of articles in a given year indicates a higher research interest in a given topic. We analyzed the research impact by the relative number of citations for all articles in a given topic and year compared to all citations for the whole year. The higher the number of citations in a topic, the higher the research impact of that topic. Fig 8 shows share charts representing the period from 1990 to 2021 (the period before 1990 had a relatively small number of articles, and the graphic results could therefore optically distort longer-term trends).

https://doi.org/10.1371/journal.pone.0268777.g008

The Fig 8 shows the development of individual topics concerning their research interest and research impact. If we take a closer look, we can identify three types of topics: rising, stable, and declining.

We have identified two topics that have a rising character. These are Topic-01 and Topic-04, for which it can be stated that in the long run, their share is growing significantly compared to other topics. This increase is particularly evident in Topic-01, which was only a marginal topic in the AHP research about ten years ago. At present, however, this topic significantly dominates both in research interest and research impact. In terms of long-term development, the increase in the share in the AHP research can also be seen in Topic-04, although this increase was more significant about five years ago. The current trend suggests that the share of articles in Topic-01 will continue to grow significantly in the coming years and will likely dominate its research impact.

The following five topics can be considered as stable topics: Topic-03, Topic-05, Topic-06, Topic-08, and Topic-09. With these topics, it can be stated that their share is relatively stable over time, both in terms of research interest and research impact. The long-term trend suggests that there will probably be no significant changes in these topics, at least in the coming years.

We have identified two topics in which research impact and research interest are declining at the time—Topic-02 and Topic-07. Interestingly, these topics have dominated in the past and have been the main scientific currents in AHP research. Gradually, however, their importance was replaced by more current topics. In addition, Topic-02 was the most important topic in the past and is currently only an average melter. An even more marked decline can be observed at Topic-07—from the second most dominant topic to the least significant. Over the period of 30 years, its significance has been reduced three times. While Topic-02 can be expected to stabilize over time, the importance of Topic-07 is likely to continue to decline.

Based on the above results, it is possible to describe the main characteristics of the nine topics. We will focus mainly on their composition, representation in the subject area, their development over time, and at the same time, we will try to briefly list several studies that can be considered significant in the given topic with regard to the number of citations.

3.3.1 Ecology & ecosystems (Topic 1).

In the first topic, there were mainly terms closely related to ecology and ecosystems. The most frequent terms in this topic were ’area’, ’water’, ’suitabl’, ’land’, ’region’ and ’potenti’. From the given composition of words, it can be concluded that the environmental focus of articles is dominant, which is directly related to ecology or ecosystem. Currently, this is a medium-sized topic (4441 articles), while from the point of view of the subject area, most articles are in the categories ENVI, EART, and AGRI. It follows that topic 1 is relatively clearly distinguishable from the others. At the same time, we can state that the development of this topic has recorded a significant growth only since 2005, not only in terms of research interest but also in terms of research impact. Our results suggest that this is the fastest-growing topic among all identified. We believe that this may be due to an intense increase in global climate problems, which is also reflected in scientific initiatives.

Several studies that have had a relatively good research impact can be pointed out in this topic. Pourghasemi, Pradhan and Gokceoglu [ 44 ] focused on the production of landslide susceptibility maps, comparing the results obtained through AHP and fuzzy logic. In an earlier study, Yalcin [ 45 ] also used AHP to create landslide susceptibility maps, using two other statistical index and weighting factor methods. The results showed slight deviations but were generally similar. Dai, Lee and Zhang [ 46 ] used AHP for geo-environmental evaluation for urban land-use planning. This is a relatively standard use of AHP as a decision-making tool that takes into account several criteria.

3.3.2 Multi-criteria decision-making (Topic 2).

The second topic contained terms such as’ decis’, criteria ’,’ select ’,’ problem ’,’ altern ’and’ rank ’. It can be seen from the nature of these terms that they are closely related to decision-making, which is why we have called this topic multi-criteria decision-making (MCDM). This topic is the most extensive of all and contains 5041 articles. At the same time, the articles contain the most citations of all topics in the entire period. According to our findings, the research impact of AHP in multi-criteria decision-making is high. Articles focused on this topic fall mainly in the subject area ENGI, COMP, BUSI, DECI. If we look at the development of this topic, it was the dominant topic of the AHP until about 2008, while later other topics began to prevail. However, multi-criteria decision-making using AHP is still the second strongest topic in the last five years. Although the research impact of this topic is slightly declining in the long run, it is still one of the largest. This is one of the most important topics, and we believe that this is because it concerns the very essence of decision-making and its objectification. At the same time, AHP is not a tool in this topic but is directly an object of research.

Tzeng and Huang [ 47 ] pointed to the use of AHP in Multiple Attribute Decision Making (MADM). AHP in their book was one of the appropriate methods in addition to TOPSIS, VIKOR, ELECTRE, PROMETEE, fuzzy integrals, and rough set theory. Rezaei [ 48 ] published a relatively successful study, presenting a new decision-making method within MCDM and calling it BWM: best-worst method. Statistical results of this study show that BWM performs significantly better than AHP with respect to the consistency ratio and the other evaluation criteria: minimum violation, total deviation, and conformity. Two articles focused on the review of MCDM methods for sustainable energy decision-making also had a relatively significant impact on this topic [ 36 , 49 ].

3.3.3 Production and performance management (Topic 3).

Terms such as ’manag’, ’product’, ’perform’, ’industri’, ’implement’ and ’organ’ formed topic 3. The composition of these terms suggests that the topic is closely related to production and performance management. This is a relatively large topic (4781 articles) with a relatively high research impact measured over the total number of citations. Most articles on this topic have been published in the subject areas BUSI, ENGI, COMP, and DECI. The topic is really closely related to production and performance management. The number of articles dealing with these topic has become more significant since about 2000. Over the last 20 years, relatively stable usage of AHP in topics related to production and performance management can be seen—this applies to the number of articles as well as research impact. We believe that despite the growing objective side of decision-making in production and performance management, there are still types of decisions in which AHP finds application. At the same time, several studies with a significant impact on this topic can be mentioned.

Sarkis [ 32 ] used AHP to assess the green supply chain management elements and how they serve as a foundation for the decision framework. Handfield et al. [ 50 ] used AHP to evaluate the relative importance of various environmental traits and assess several suppliers’ relative performance along with these traits. Seuring [ 51 ] focused on sustainable supply chain management models, identifying AHP as one of the relevant approaches.

3.3.4 Sustainable development (Topic 4).

In the fourth topic related to AHP, the most commonly used terms were ’develop’, ’sustain’, ’environment’, ’resource’, ’econom’ and ’environ’. The first term dominated, while the representation of others was additional information. Given the composition of these terms, the topic was named sustainable development. This is a smaller topic (3205 articles), while the total number of citations of these articles is also smaller. Most articles focused on sustainable development were from the subject areas ENVI, ENGI, SOCI, and AGRI. Until 2015, this was a relatively insignificant topic, but it has grown since 2016 and is currently one of the relevant areas with the use of AHP. As with the first topic, it can be deduced that the increase in articles focused on sustainable development and AHP is related to the increase in scientific interest in environmental topics.

Brandenburg et al. [ 52 ] literature review in which he focused on quantitative models for sustainable supply chain management. In this study, he analyzed 134 papers, identifying that AHP is one of the most widely used methods of SCM-related decision making. Wu and Webster [ 53 ] used AHP as part of a multi-criteria evaluation simulation of land development. The suitability of AHP as a tool for comprehensive Environmental Impact Assessment—EIA, for example, was analyzed by Ramanathan [ 40 ], who addressed its benefits and described its shortcomings.

3.3.5 Computer networks, optimization and algorithms (Topic 5).

The fifth topic consisted mainly of terms such as ’design’, ’network’, ’optim’, ’base’, ’time’ and ’algorithm’. This topic was more heterogeneous in terms of meaning, so we chose its broader name—computer networks, optimization and algorithms. This is a medium-sized topic (4191 articles), and the total number of citations to these articles was small compared to other topics. Most articles on this topic have been published in the subject areas ENGI, COMP, MATH, and PHYS. Interest in this topic has been only marginal in the past, with a slight increase in the number of articles since about 2015. Studies published between 2002 and 2013 had the highest research impact. Compared to other topics, the ratio between research interest and research impact in this fifth topic is unfavorable—the number of articles is higher, but the number of citations is lower. This may be due to the fact that computer science or mathematics is an exact science and has more suitable tools such as AHP to solve scientific problems.

Song and Jamalipour [ 54 ] published a study in which AHP was used to decide the relative weights of evaluation criteria set according to user preferences and service applications as a base to rank the network alternatives. Lin et al. [ 55 ] focused on customer-driven product design, using AHP to evaluate the relative overall importance of customer requirements and design characteristics. Mouzon and Yildirim [ 56 ] used AHP to determine the ’best’ alternative among the solutions on the Pareto front.

3.3.6 Service quality (Topic 6).

Articles in the sixth topic had terms such as ’qualiti’, ’servic’, ’import’, ’expert’, ’provid’ and ’inform’. Such terms are semantically most associated with the field of service quality, so we named the sixth topic this way. The sixth topic is medium in size (3670 articles), and its research impact is smaller (articles of this topic are not on average significantly cited compared to articles from another topic). Representation of service quality was in practically all subject areas, but the four most important are ENGI, SOCI, COMP, and MEDI. Topic service quality is the most stable topic in terms of time development—practically since 1994, it has been steadily equally represented and thus undoubtedly forms an important topic with history and current applications. The stability of the sixth topic was not only recorded in terms of the number of articles but also in terms of the number of citations. Given the global development of society and the transformation of many economies in terms of services, it is possible to see continuing interest in services and their quality. This could partly explain the above characteristics of this sixth topic.

The following three studies can be included among the most important studies. Cheever et al. [ 57 ] focused on prioritizing cancer antigens in a medical study and used AHP to deal with complex decisions. The second study is by Ho [ 13 ], which focused on the analysis of articles in which AHP is combined with other tools such as QFD, DEA, or SWOT, stating that integrating AHP with other methods is generally better than stand-alone AHP. A work by Forman and Gass [ 8 ] focused on exposing the reasons for AHP’s wide variety of applications and the efficacy and applicability of the AHP vis-a-vis competing methodologies.

3.3.7 Fuzzy logic (Topic 7).

The seventh topic consisted of terms such as ’weight’, ’fuzzi’, ’valu’, ’determin’, ’obtain’ and ’base’. The first two terms dominated this topic, so we named it fuzzy logic. The size of the topic is smaller (2991 articles), but its research impact is higher compared to other topics. We believe that a small number of articles contain important information that is applicable to various subject areas. Articles focused on AHP and fuzzy logic were mainly from the subject areas ENGI, COMP, MATH and DECI. Given the time evolution of the topic, Fuzzy logic played a very important role in the AHP method, especially in the nineties. Research interest and research impact of fuzzy logic steadily declined. Nevertheless, it cannot be said that this is a dying topic. The "decline" of this topic is due to the faster growth of other topics, while fuzzy logic still finds significant applications in various subject areas.

One of the popular studies by Mikhailov [ 58 ] focused on deriving priorities from fuzzy pairwise comparison judgments is proposed, based on α-cuts decomposition of the fuzzy judgments into a series of interval comparisons. Six years later, Wang et al. [ 36 ] proposed extent analysis method on fuzzy AHP to obtain a crisp priority vector from a triangular fuzzy comparison matrix. They found that the extent analysis method cannot estimate the true weights from a fuzzy comparison matrix and has led to quite a number of misapplications in the literature. Another important study from Alonso and Lamata [ 59 ] presented a statistical criterion for accepting/rejecting the pairwise reciprocal comparison matrices in the analytic hierarchy process.

3.3.8 Systematic evaluation (Topic 8).

We named the eighth topic systematic evaluation because it contained two dominant terms, ’evaluation’ and ’system’. In addition, other terms have been identified that can be considered complementary—’energy’, ’indic’, ’establish’ and ’power’. This is a medium-sized topic (3893 articles) with less research impact. More than a third of all articles (37%) on this topic were published in the subject area ENGI, followed by less represented areas such as ENER, COMP, and ENVI. This may be partly logical, as such links have already been shown to us in the previous analysis in Chapters 3.1 and 3.2. By the year 2000, this topic was practically negligible, but it gradually began to grow, and even in the period 2009–2014, it was one of the three most important. The growth of this topic around 2015 has stabilized and currently has approximately the same proportion of articles focused on AHP. Research interest exceeds research impact, which is comparable to less important topics. Nevertheless, there are several studies whose research impact has been relatively significant.

By far, the most significant research impact was recorded by an article by Thomas Saaty [ 19 ], author of AHP, who published a summary study on AHP. He presented principles and the philosophy of theory, and he summarized general background information of the type of measurement utilized, its properties, and applications. The impact of this study was enormous (4785 citations) and significantly affected a number of other subject areas. None of the other studies on this topic came close to the impact of Saaty’s work. This would partly explain the small research impact of the topic of systematic evaluation at present—we assume that current research refers to the original article published in 1990. The second study with much less impact—but not negligible, given the 383 citations—is by San Cristóbal [ 60 ]. In it, the author focuses on using AHP for weighting the importance of different criteria, which allows decision-makers to assign these values based on their preferences. Hermann, Kroeze and Jawjit [ 61 ] published a study in which they presented a new analytical tool, called COMPLIMENT, based on AHP, which can be used to provide detailed information on the overall environmental impact of a business.

3.3.9 Risk assessment (Topic 9).

The last topic was formed by the terms ’risk’, ’assess’, ’project’, ’construct’, ’safeti’ and ’structur’. Since the first two terms dominated, we called this topic risk assessment. In terms of numbers, this topic is small (3218 articles) and has a correspondingly small research impact. The articles in this topic are from all research areas, but they are significantly dominated by the subject area ENGI (36%), followed by ENVI (7%), EART (7%), and COMP (7%). We assume that risk assessment dominates the most in ENGI, but it is relevant for practically all subject areas. The risk assessment topic has been relatively stable since about 2000 if we take into account the research interest. The research impact of this topic has been relatively stable since 2000. Studies with the highest impact can also be mentioned in this topic.

Esawi and Farag [ 62 ] used AHP to select the optimum material for a tennis racket. AHP was used in the decision-making phase, in which it was necessary to eliminate subjectivity and thus reduce the risk of a wrong decision. Yüksel and Daǧdeviren [ 63 ] also published a study using AHP in SWOT analysis. They state that although the AHP technique removes these deficiencies, it does not allow for the measurement of possible dependencies among the factors and therefore, they propose their own algorithm using ANP (analytical network process). The third major study was published by Kutlu and Ekmekçioǧlu [ 64 ] directly used the risk assessment tool—Failure mode and effects analysis (FMEA). In this study, a fuzzy approach was developed. It allows experts to use linguistic variables for determining S, O, and D, by applying fuzzy TOPSIS integrated with fuzzy AHP.

4 Discussion

4.1 summary of main findings.

We presented the main results of processing an extensive dataset of scientific documents in sections 3.1, 3.2 and 3.3. In the introduction to the article, we have defined three research questions, to which we will now try to find brief answers. At the same time, it should be noted that a more comprehensive answer to the questions can be found in Chapter 3.

RQ1: What is the usage of AHP and research impact in individual subject areas? Main highlights:

• The most represented areas of AHP use clearly include Engineering (ENGI 25.6%), followed by Computer Science (COMP 12.0%), Environmental Science (ENVI 10.3%), and Business, Management and Accounting (BUSI 8.4%).
• Publications on AHP achieve the highest research impact in the subject areas Decision Sciences (38.8 citations per article) and Mathematics (28.8 citations per article). This could be explained by the fact that these areas have been the basis for the development of AHP in the past, and so far, the authors deal with the very essence of AHP.

RQ2: What are the trends in the popularity of AHP from the first introduction of the method in 1980 to the present? Main highlights:

• The publication of AHP articles is growing very significantly over time . In the last four years (2017–2021), more than 15,000 new articles with such a focus have been published.
• The highest increase in the total number of articles concerned the subject area Environmental Science —by 2010, 582 articles had been published, and since 2010 it was 3,623 articles, which represents a more than 6-fold increase. For comparison, the most numerous subject area Engineering recorded an "only" 3.5-fold increase. With current trends, it can be expected that in 3 to 4 years, the subject area Environmental Science could already be the most numerous area in articles related to AHP.
• Interest in AHP has grown significantly among researchers in the subject area of Business , Management , and Accounting —currently, one in 300 articles published in this field is focused on AHP (in 2005, it was 1 article in 1000 published). This topic has been consistently popular for almost two decades in Engineering (1 article in 250 published) and Decision Sciences (1 article in 300).

RQ3: What are the most common topics related to AHP, and what is their development over time? Main highlights:

• ■ Ecology & Ecosystems . A relatively new topic, probably related to the growing interest in environmental issues in the world; it has been growing significantly since about 2005.
• ■ Multi-criteria decision-making . The most extensive topic in which the authors deal with the very essence and improvements of the AHP method.
• ■ Production and performance management . A stable topic focused on the application of AHP to various aspects of managerial decision-making related to production and performance.
• ■ Sustainable development . A topic with a rapid growth rate, which can be explained by the increasing intensity of sustainability research.
• ■ Computer network , optimization and algorithms . A relatively heterogeneous topic that uses AHP to objectify decisions that other mathematical apparatuses cannot solve.
• ■ Service quality . An extremely stable topic in which AHP is used in various aspects of service quality research.
• ■ Fuzzy logic . The topic was dominant and showed a high research impact in the last century dealing with weights determination using fuzzy logic.
• ■ Systematic evaluation . Practically oriented topic with less research impact focused on the use of AHP in technically oriented decisions, especially in the field of Engineering.
• ■ Risk assessment . A smaller but stable topic covering themes related to the use of AHP in risk assessment in various application areas.
• The highest increase in the share in terms of time development was recorded by the topic Ecology & Ecosystems . This applies to both research interest (number of articles) and research impact (number of citations).
• Articles related to AHP differ depending on the research object . Multi-criteria decision-making and Fuzzy logic are two topics that deal with the very essence of AHP—principles, axioms, rules, and development—and AHP is directly the subject of their research. The other seven topics are used by AHP primarily as a tool for other various research objects.

A summary of the individual characteristics of the identified topics can be found in Table 2 .

https://doi.org/10.1371/journal.pone.0268777.t002

4.2 Theoretical and practical implications in production research

Our overview of the use of AHP offers a general picture of this universal method in different subject areas and different topics. If we take a closer look at studies that are directly focused on production research, we could identify three main areas.

The first area is the use of AHP in supply chain management. With regard to the composition of articles on SCM, it can be fairly argued that this is a top area with the use of AHP in the field of production research. This was confirmed, among other things, by an overview of the three most cited articles in the third topic, Production and Performance Management. The AHP is useful in SCM, especially if the research focuses on green SCM [ 74 – 78 ] or supplier evaluation or selection [ 33 , 35 , 79 – 85 ].

The second area is multi-criteria decision-making. Apart from the fact that this topic was also identified by our analysis, after a deeper examination of the articles in production research, we see a similar use. For example, Bhattacharya, Sarkar and Mukherjee [ 86 ] combined AHP with QFD to select a robot. This selection was based on requirement analysis, and AHP plays a role in weight determination requirements. Wei, Chien and Wang [ 87 ] also used AHP in 2005 to support ERP (Enterprise Resource Planning) selection decisions. In the later period, more advanced modifications of the AHP were used in production research decisions. Bouzon et al. [ 88 ] used fuzzy AHP to analyze reverse logistics barriers. Achieving optimal decision-making of cloud manufacturing service provided was the subject of a study by Hu et al. [ 89 ], while its authors used, in addition to AHP, other more advanced decision-making tools such as TOPSIS or Grey Correlation Analysis. AHP has also been used to support decision-making for logistics operations in distribution centers [ 90 ]. Last but not least, Ishizaka et al. [ 91 ] used AHP in conjunction with Data Envelopment Analysis (DEA) to multi-criteria inventory classification. It can be seen from this overview that AHP combines relatively well with other tools, whether they are requirements-based tools (such as QFD) or decision support tools (such as TOPSIS).

The third area in production research where AHP applications can be found is risk. We also identified this topic in our analysis. If we take a closer look at the articles focused on risk management or risk assessment, we can see a relatively wide range of applications. Samvedi, Jain and Chan [ 92 ] used fuzzy AHP and fuzzy TOPSIS to quantify risks in a supply chain. Dong and Cooper [ 93 ] pointed to the fact that the traditional assessment methodologies are unable to deal with intangible criteria, which are a crucial factor in the analysis. They developed an orders-of-magnitude AHP (OM-AHP) based ex-ante supply chain risk assessment model to compare the tangible and intangible elements that influence supply chain risks. Ilbahar et al. [ 94 ] used Pythagorean fuzzy AHP & fuzzy inference system to risk assessment for occupational health and safety, comparing the results with another risk assessment tool—FMEA (Failure Mode and Effect Analysis). Kumar et al. [ 95 ] use fuzzy AHP to prioritize the risks under vague and unclear surroundings.

In addition to these theoretical benefits, our research may have several practical implications. One of the most significant practical findings is that AHP is a truly universal decision-making tool, documented by more than 40 years of research. The use of AHP to objectify the work of decision-makers in the industry can have several levels—basic, advanced, and expert. At the basic level—for managers who do not have much experience with the systematic assessment of unstructured problems, the basic version of the AHP can already be a functional tool for qualitative-quantitative decision-making. The systematic evaluation was also one of the identified topics, while it was very widely represented almost in all subject areas—which testifies to the universality of AHP. At the advanced level—for managers who know and use simpler and moderately demanding decision support tools, AHP can help assess criteria through a multi-criteria decision-making process. In such cases, the AHP acts as a support tool, usually a multi-step decision-making tool. At the expert level—AHP can also be used in complex production systems to increase productivity, reduce risk or objectify strategic decisions. Solutions related to fuzzy logic can serve in such types of decisions, and even small improvements can bring significant economic and non-economic benefits in complex production systems.

4.3 Research limitations and future research opportunities

Several research limitations can also be identified in our research. Non-absolute indexing, which is the first limitation, refers to sampling bias due to the limitations of the Scopus database. The Scopus database does not index all scientific articles related to the AHP method. Some AHP-related articles may only be exclusive to the Web of Science or other databases. However, several studies suggest a significant overlap between the Scopus and Web of Science databases [ 96 , 97 ]. At the same time, Scopus contains more than 76 million records, making it the world’s largest abstract and citation database. The sample of articles in this paper was very high (more than 34,000 documents), and from a statistical point of view, this can be considered as a representative and robust solution. At the same time, it should be noted that different databases (e.g., Scopus, Ebsco, Web of Science) have data in different structures—e.g., they have different subject areas. By combining databases, we would probably achieve a higher number of articles, but their data structure would not be consistent, and therefore we would not be able to answer the first two research questions objectively.

The second limitation is the partial inclusion of incorrect articles caused by synonymous terms. However, we assume that this proportion of articles was small enough and did not significantly affect the results of our research.

The third limitation is limited text analysis–we analyzed only abstracts of articles. On the other hand, we assume that this shortcoming was not major because the abstract usually contains the most relevant information. The median number of processed words in the abstract was 185.

The fourth limitation is the absence of a complete PRISMA methodology used for articles such as systematic literature reviews. It should be noted that our article is not a standard type of systematic literature review article. Our article does not focus on in-depth analysis of a limited number of relevant articles but uses big data approaches and machine learning tools to cover the topic of AHP as comprehensively as possible. It is important to emphasize that study is not a standard type of systematic literature review. We do not focus on a specific issue in a specific area of interest—we focus on a comprehensive overview of AHP in all areas. Such approaches to literature reviews are currently beginning to be applied to a number of rapidly evolving topics [ 98 , 99 ].

Our approach offers also several opportunities for further research. First of all, a multistage analysis of topics can be mentioned. We analyzed the entire dataset of documents when identifying topics, which helped us identify the most prominent "macro" topics. Each topic could be subjected to further analysis to identify more detailed "micro" topics. This could contribute to a better understanding of the development of AHP and its use in scientific work.

Another potential is the application suitability of LDA and the robustness of its results. According to current data from the Scopus database, more than 5,000 documents concern Latent Dirichlet Allocation, of which approximately 11% also contain the keywords “review”. In most cases, LDA was used to model topical practice-oriented topics. The use of LDA for the analysis of topics in online reviews was used by Guo, Barnes and Jia [ 100 ] or Tirunillai and Tellis [ 101 ]. Calheiros, Moro and Rita [ 102 ] used LDA to gather relevant topics that characterize a given hospitality issue by a sentiment. Boussalis and Coan [ 103 ] focused on analyzing the signals of climate change doubt, using the LDA on more than 16,000 documents from 19 organizations between 1998 and 2013.

The LDA has only been used in recent years to analyze topics in research areas. D’Amato et al. [ 104 ] used bibliometric data to analyze the green, circular, and bioeconomy areas. Mäntylä, Graziotin and Kuutila [ 105 ] used the evolution of sentiment analysis to analyze nearly 7,000 documents from the Scopus database. According to some sources, unstructured data (eg text) represents more than 80% of all data [ 106 ]. Thus, LDA appears to be a suitable method for research on topics, enabling it to cover a large number of documents and extract relatively meaningful and interpretable results. In this article, we have focused on the analysis of the use of the AHP method, but LDA can be applied to virtually any research theme in which the analysis of topics is relevant. We assume that the use of LDA for topic analysis in various research areas will grow.

5 Conclusion

There are a large number of AHP applications, and it is quite difficult to capture them in all their complexity. Although standard literature reviews offer an up-to-date view of the most important publications, they naturally cannot cover the topic in its entirety. Our approach to literature review-based LDA topic modeling is the first to be used on the AHP. This probabilistic clustering approach makes it possible to process a large amount of information and identify the most frequent topics in the corpus of documents.

In our study, we analyzed more than 35,000 abstracts of scientific documents from the Scopus database related to AHP. We cover three research questions with our analysis. The first was to identify the usage of AHP and its research impact in individual subject areas. In the second research question, we focused on the analysis of trends in the popularity of AHP from the first introduction of the method in 1980 to the present. We analyzed the trends with regard to individual subject areas. The third research question was to capture the time evolution of AHP-related topics. We identified nine topics, which we subjected to a deeper statistical survey, and we captured the development over time with regard to the research interest and research impact of each topic.

Given the long-term growth trend of articles focused on or using AHP, our results can offer an up-to-date and robust information base for further research. We believe that our study can provide a basis for a broader scientific discussion on AHP. We also believe that the use of topic modeling has great potential in the literature review in any research area.

Acknowledgments

We would like to thank the reviewers for their careful reading of our manuscript and their many insightful comments and suggestions.

• 1. Saaty TL. The Analytic Hierarchy Process. 1980. New York: Mc-Graw-Hill.
• View Article
• Google Scholar
• PubMed/NCBI
• 31. Chuang J, Manning CD, Heer J. Termite: Visualization techniques for assessing textual topic models.” In Proceedings of the international working conference on advanced visual interfaces, edited by Tortora Genny, Levialdi Stefano, Tucci Maurizzio, 74–77. 2012. Association for Computing Machinery: New York.

Academia.edu no longer supports Internet Explorer.

To browse Academia.edu and the wider internet faster and more securely, please take a few seconds to  upgrade your browser .

•  We're Hiring!
•  Help Center

Analytical Hierarchy Process (AHP)

• Most Cited Papers
• Most Downloaded Papers
• Newest Papers
• Save to Library
• Last »
• Decision Making Analysis and Modeling Follow Following
• Decision Analysis Follow Following
• Delphi Methodology Follow Following
• Construction Industry Follow Following
• Multiple criteria decision making (MCDM) Follow Following
• SPSS Software Follow Following
• Fuzzy AHP Follow Following
• AHP green supply management Follow Following
• TOPSIS Follow Following
• Green Building Materials (Architecture) Follow Following

Enter the email address you signed up with and we'll email you a reset link.

• Academia.edu Publishing
•   We're Hiring!
•   Help Center
• Find new research papers in:
• Health Sciences
• Earth Sciences
• Cognitive Science
• Mathematics
• Computer Science
• Academia ©2024

Applying the Analytic Hierarchy Process in healthcare research: A systematic literature review and evaluation of reporting

Affiliations.

• 1 Center for Health Economics Research Hannover (CHERH), Leibniz University of Hanover, Otto-Brenner-Str. 1, 30159, Hannover, Germany. [email protected].
• 2 Center for Health Economics Research Hannover (CHERH), Leibniz University of Hanover, Otto-Brenner-Str. 1, 30159, Hannover, Germany. [email protected].
• 3 Institute for Risk and Insurance, Leibniz University of Hanover, Otto-Brenner-Str. 1, 30159, Hannover, Germany. [email protected].
• 4 Center for Health Economics Research Hannover (CHERH), Leibniz University of Hanover, Otto-Brenner-Str. 1, 30159, Hannover, Germany. [email protected].
• 5 Center for Health Economics Research Hannover (CHERH), Leibniz University of Hanover, Otto-Brenner-Str. 1, 30159, Hannover, Germany. [email protected].
• PMID: 26703458
• PMCID: PMC4690361
• DOI: 10.1186/s12911-015-0234-7

Background: The Analytic Hierarchy Process (AHP), developed by Saaty in the late 1970s, is one of the methods for multi-criteria decision making. The AHP disaggregates a complex decision problem into different hierarchical levels. The weight for each criterion and alternative are judged in pairwise comparisons and priorities are calculated by the Eigenvector method. The slowly increasing application of the AHP was the motivation for this study to explore the current state of its methodology in the healthcare context.

Methods: A systematic literature review was conducted by searching the Pubmed and Web of Science databases for articles with the following keywords in their titles or abstracts: "Analytic Hierarchy Process," "Analytical Hierarchy Process," "multi-criteria decision analysis," "multiple criteria decision," "stated preference," and "pairwise comparison." In addition, we developed reporting criteria to indicate whether the authors reported important aspects and evaluated the resulting studies' reporting.

Results: The systematic review resulted in 121 articles. The number of studies applying AHP has increased since 2005. Most studies were from Asia (almost 30%), followed by the US (25.6%). On average, the studies used 19.64 criteria throughout their hierarchical levels. Furthermore, we restricted a detailed analysis to those articles published within the last 5 years (n = 69). The mean of participants in these studies were 109, whereas we identified major differences in how the surveys were conducted. The evaluation of reporting showed that the mean of reported elements was about 6.75 out of 10. Thus, 12 out of 69 studies reported less than half of the criteria.

Conclusion: The AHP has been applied inconsistently in healthcare research. A minority of studies described all the relevant aspects. Thus, the statements in this review may be biased, as they are restricted to the information available in the papers. Hence, further research is required to discover who should be interviewed and how, how inconsistent answers should be dealt with, and how the outcome and stability of the results should be presented. In addition, we need new insights to determine which target group can best handle the challenges of the AHP.

Publication types

• Research Support, Non-U.S. Gov't
• Systematic Review
• Decision Making*
• Health Services Research* / methods
• Health Services Research* / standards
• Health Services Research* / statistics & numerical data

Performance indicators and analytic hierarchy process to evaluate water supply services management in Algeria

• Article contents
• Figures & tables
• Supplementary Data
• Open the PDF for in another window
• Guest Access
• Cite Icon Cite
• Permissions
• Search Site

Sofiane Boukhari , Dounia Mrad , Sabri Dairi; Performance indicators and analytic hierarchy process to evaluate water supply services management in Algeria. AQUA - Water Infrastructure, Ecosystems and Society 2024; jws2024040. doi: https://doi.org/10.2166/aqua.2024.040

Download citation file:

• Ris (Zotero)
• Reference Manager

The provision of an efficient water supply service (WSS) is crucial for the well-being of citizens and the sustainability of cities. This study aims to evaluate the performance of WSS using the results of a household survey and the ranking of performance indicators (PIs) by the analytic hierarchy process method. The methodology developed was tested for the case of the city of Taoura (Algeria). A survey was carried out among 340 residents of the city. The survey results showed that the majority of respondents (70%) were relatively dissatisfied with the quantity of water provided and 67% of households surveyed rated the quality of service as poor. Then, the performance was evaluated according to 5 decision criteria and 20 PIs. The results of the evaluation of the relative weights of the criteria are as follows: the ‘Financial and economic’ criterion plays the most important role, with a relative weight of 38.61%, followed by the ‘Operational’ criterion (24.7%) and the criterion ‘Physics’ (17.32%). The methodology used in this study can be a reliable tool for evaluating the performance of WSS in developing countries.

Development of a methodological framework to identify and classify performance indicators.

Water supply services are facing issues in achieving sustainability objectives.

Five criteria and 20 performance indicators were identified for assessment.

Weights were generated using the analytic hierarchy process.

Calculate the benefits for households and water companies with the new water tariff estimated by the WTP.

Supplementary data

Affiliations

• ISSN 2709-8028 EISSN 2709-8036
• Open Access
• Collections
• Subscriptions
• Subscribe to Open
• Editorial Services
• Rights and Permissions
• Sign Up for Our Mailing List
• IWA Publishing
• Republic – Export Building, Units 1.04 & 1.05
• 1 Clove Crescent
• London, E14 2BA, UK
• Telephone:  +44 208 054 8208
• Fax:  +44 207 654 5555
• IWAPublishing.com
• IWA-network.org
• IWA-connect.org
• Cookie Policy
• Terms & Conditions
• Get Adobe Acrobat Reader
• ©Copyright 2021 IWA Publishing

This Feature Is Available To Subscribers Only

Sign In or Create an Account