A systematic review: the dimensions to evaluate health care performance and an implication during the pandemic

The inclusion and exclusion criteria were set as shown in Table 1 based on the PICO (Population, Intervention, Comparison, and Outcome) tool [44]. Additionally, all study designs were included.

First, a search strategy was developed for the PubMed database (see Table 1). Then, the strategy design was adapted for the Embase, Cochrane CENTRAL, and Google Scholar databases. Furthermore, an additional search in the Google engine was performed to find gray literature or unpublished papers, including theses and conference abstracts. Additionally, the reference lists of all the eligible articles were checked. The databases were searched from inception until October 2020.

The search strategy was developed by the first, second, and fourth authors; the first two were experts in health care management and BSC, while the fourth author was an expert in systematic reviews and meta-analysis. It was initially developed for the PubMed database. Moreover, the search strategy was developed to include both Medical Subject Headings (MeSH) terms and keywords to widen the search frame and obtain more results. Then, appropriate truncation and relevant indexing terms were used. See Appendix (S2), which shows search strategies in all databases. Afterward, the first author conducted an electronic database search and removed duplicates using the EndNote X9.2 program.

The selection of eligible studies was performed independently by the first and second authors in all steps. Disagreements were resolved by discussion after each step or, if necessary, through arbitration by the fourth author. First, the articles' titles and abstracts were examined to eliminate irrelevant papers between November 2020 and February 2021. Then, full texts were carefully inspected to decide the final papers' inclusion list between February and June 2021. If different KPIs were used in more than one implementation in the same study, each was counted as a different implementation. In comparison, implementations using the same KPIs in other locations or times in the same research were considered one implementation. The authors of studies with no available full texts or with partially reported results were contacted for missing data.

Data extraction was performed between June and July 2021 and then compared to discuss differences. The following data were extracted from the eligible studies: 1) author/s, 2) year of publication, 3) country of origin, 4) data collection duration, 5) data collection tool, 6) the number of perspectives, 7) the number of KPIs, 8) availability of weights/importance for perspectives or KPIs, and 9) outcome, which is represented in the KPIs that have been used and their weights/importance. The frequency of each KPI used at each implementation was plotted on Microsoft Excel, and the sum was calculated. In addition, the weight/importance assigned for each KPI at each implementation was reported on a scale of 100%. In the case of studies that did not give weights/importance explicitly, each KPI weight/importance was calculated by dividing one by the number of KPIs used in that study to assign an equal weight/importance for each KPI.

Consequently, we computed an average of the weights/importance assigned for each KPI. Next, we performed regrouping and coding for the KPIs to find the frequency of use and the set weights/importance percentages for each dimension. Then, the resulting major and subdimensions were listed and described between August and September 2021.

The research design of eligible studies was extracted directly from the studies. However, if the research design was not explicitly mentioned, we determined it based on the role of the investigator in that study. Specifically, the study was considered observational if the BSC exposures were naturally determined and the investigator had no part. On the other hand, the study was considered experimental if the investigator actively assigned the BSC intervention.

The risk of bias (RoB) assessment was accomplished between August and September 2021 to assess the quality of the included studies. The Risk of Bias in Nonrandomized Intervention Studies (ROBINS-I) tool was used to evaluate the observational and quasi-experimental studies [45]. In comparison, the Cochrane (RoB 2) tool was used for the assessment of randomized controlled trials (RCTs), as per the Cochrane collaboration's guidelines [46]. The RoB was analyzed at the study level and across studies since authors should avoid summarizing the overall RoB as per the Cochrane Handbook guidelines [47, 48].

In the RoB 2 tool, five types of bias were assessed: bias arising from the randomization processes, bias due to deviations from intended interventions, bias due to missing outcome data, bias in the measurement of outcomes, and bias in the selection of the reported results. On the other hand, in the ROBINS-I tool, seven types of bias were assessed: bias due to confounding, bias in the selection of participants in a study, bias in the measurement/classification of interventions/exposures, bias due to deviations from intended interventions/exposures, bias due to missing data, bias in the measurement of the outcomes, and bias in the selection of the reported results.

Using the RoB 2 tool, each type of bias was assessed as low, high, or unclear. While using the ROBINS-I tool, each type of bias was evaluated into five categories: low, moderate, serious, critical, or no information. Figures for RoB were prepared using the ROBVIS (Risk Of Bias VISualization) tool [49]. Last, it was recommended not to advocate quality appraisal as a criterion for inclusion in reviews [50]. Therefore, the authors decided to include all studies in this systematic review regardless of their quality assessment. See Quality assessment below.

A total of 4028 studies resulted from running the search strategy in the four databases. In addition, another three studies were identified through a Google search. Therefore, a total of 4031 studies were included. Duplicates were removed (n = 1046) using the EndNote program, and then the remaining articles were screened based on their titles and abstracts (n = 2985). Irrelevant papers were excluded (n = 2794).

Consequently, the remaining 191 studies were examined by reading the full texts. Among these papers, 22 papers were written in non-English languages, including Spanish, German, French, Chinese, and Persian. A full-text translation was performed for each study to decide whether to include or exclude any of them. As a result of reading the full texts, 158 studies were excluded, and only 33 were eligible for this review, in which 36 full implementations of different BSC designs were actually applied. Table 2 shows a summary of the 36 implementations. Details of the study selection process are shown in the PRISMA flowchart [43]. See Fig. 3.

Some of the resulting studies integrated multiple-criteria decision analysis (MCDA) with BSC. One study [82] combined BSC with simulation and MCDA techniques with what was referred to as S-MEDUTA. Another study [60] integrated the BSC with fuzzy analysis. Two studies [67, 78] combined BSC with AHP, and one [78] used the TOPSIS technique. Studies explained that using these methodologies with the BSC would help them arrive at more informed and better decisions.

A total of 797 KPIs were extracted from the resulting implementations. These KPIs were categorized in the studies under 15 perspectives. The average number of perspectives used per study was 4.5, and for the KPIs, it was 22. The most frequently used perspectives were the internal, financial, patient, learning and growth, HCW, managerial, community, and stakeholder perspectives. The total use frequencies of these perspectives at the implementations were 29.6%, 17%, 12.6%, 12.6%, 9.4%, 6.3%, 5%, and 3.1%, respectively. On the other hand, the topmost important perspectives from the health managers' viewpoint were the internal, financial, learning and growth, patient, HCW, community, managerial, and stakeholder perspectives with a total weight/importance of 37.9%, 15.4%, 12%, 11.3%, 7.8%, 7.7%, 3.6%, and 2.8%, respectively.

The 797 extracted KPIs were plotted according to their frequencies and weights/importance in the categorization process. See Appendix (S3). Grouping and recategorizing KPIs resulted in Figs. 4 & 5 below, showing 13 major dimensions and 45 subdimensions based on their.

Frequency of use and importance, respectively. After regrouping these KPIs into homogenous major dimensions and subdimensions, 13 major dimensions resulted, with 45 subdimensions. A summary of the resulting perspectives and their major and subdimensions contents are illustrated in Fig. 6. The description of each major and subdimensions is described further in Appendix (S4).

Each study was evaluated in terms of RoB, as illustrated in Appendix (S5). The RoB 2 tool was utilized to assess the ROB in the sole RCT study [72], for which the assessment showed fair evaluation, except for performance bias. On the other hand, utilizing the ROBINS-I tool for assessing the RoB in observational and quasi-experimental studies revealed no information about confounders' adjustment methods except in three studies [51, 59, 67]. The confounding agents were apparent in the three studies; one study [51] performed confounders adjustments. On the other hand, another [59] adjusted for patient severity but not for the LOS and mortality rate. Last, one study [67] did not perform adjustment at all, which may have affected measurement precision.

The selection bias across studies reflected a serious RoB in five studies [59, 63, 67, 73, 80]. Therefore, the intervention and the follow-up did not coincide, and a potentially substantial amount of follow-up was missing in their analysis. Studies with a moderate risk of intervention/exposure measurement bias reflected a well-defined intervention status, but some aspects of the assignments of intervention status were determined retrospectively. Furthermore, bias in selecting the reported results was serious in one study that partially reported the results [80]. Studies that reported all results but did not have a preregistered protocol or whose outcome measurements were not defined in an initial plan were given a moderate risk. See Appendix (S5).

All the perspectives, dimensions, and KPIs employed in BSC implementations were collected to fulfill the research aims. Categorization and regrouping of the KPIs into major and subdimensions were performed. Then, the dimensions were ranked according to their frequency of use and their importance. The BSC tool can offer comprehensive planning, monitoring, evaluation, and improvement of HCO KPIs. Hence, their performance should be improved in the short and long term.

In general, studies had either no information or low or moderate ROB. At the same time, only a few of them had serious or critical ROB. However, studies that had only fully reported BSC indicator measurements were included. Many of them did not have a preregistered protocol or predefined measures in their plan. No information was found regarding confounders or deviation from unintended interventions.

Analyzing the results shows that BSC implementations typically utilized four fundamental perspectives: financial, customer, internal, and knowledge and growth. However, the analysis of Fig. 5 revealed the frequent employment and the importance of other BSC perspectives in BSC implementations. Specifically, the external and managerial perspectives. This reflects the need for slight modifications of BSC design and corresponds with the findings of another study [87], which referred to the sustainability perspective of the BSC as the fifth pillar. Additionally, our findings reveal that focusing on both internal and external customers from the customer perspective is essential.

The variation among BSC implementations in the categorization of the same KPIs reflects the need for data standardization. HCW training-related KPIs, for example, were categorized under the learning and growth perspective in almost half of the resulting studies [58‐60, 62, 64‐78, 84]. Meanwhile, the rest of the studies categorized them under the perspectives of HCW [72, 76], quality [54], service capacity, provision/service capacity [64, 68, 70, 73, 76, 84], and healthcare facility functionality [75]. These results are consistent with a study [5] that referred to the lack of defining measures and the lack of data standardization. The differences in categorization prove our assumptions in the calculation imprecision in the previous reviews. Specifically, in the use frequency or the importance of the perspectives and KPIs. Our systematic review solved this calculation bias by uniformly forming the 797 KPI categorizations. Regrouping similar or semisimilar KPIs under the same category resulted in more precise results. Unification of dimensions can guide uniform future implementations of PE or BSC at HCO, allowing data sharing and comparability. Dimension unification can be why our findings are different from another systematic review [88] that did not consider unifying the classification of KPIs. According to HCO management, the average LOS, HAIs, patient satisfaction, bed occupancy, and bed turnover rate were the most useful KPIs.

Analyzing the results also shows a lack of BSC utilization in HCO during the pandemic. Additionally, there has been a lack of studies comprehensively examining the impact of COVID-19 on KPIs.

This review can guide healthcare managers and researchers since the resulting dimensions can be utilized to synthesize future BSC measurements. Specifically, the dimensions can direct the creation of new instruments to engage stakeholders in future BSC implementations. Moreover, this review can provide a road map for healthcare managers to perform a comprehensive PE of HCO during the COVID-19 pandemic. Since the COVID-19 pandemic may influence the BSC dimensions positively or negatively compared to prepandemic, analyzing the effect of the pandemic on the performance of the major- and subdimensions will allow HCO’s managers to better understand where to focus on their action plans to improve the overall performance of HCOs.

Although this systematic review included ten months after the initiation of the COVID-19 pandemic, no research on BSC utilization in COVID-19 was found. Moreover, health policy experts stated that insufficient standardization of quality measurement approaches in the COVID-19 era challenged sharing purposes. As a result, the comparison between the performance of healthcare systems is disrupted [5]. Comparison is critical in cases where the optimal performance is not fully understood as in pandemics, and a comparison with other health systems would be informative and necessary [5]. Therefore, addressing the lack of data standardization was suggested to be overcomed by quickly defining measures, which could allow health systems, at least in the short term, to use standardized methods to better understand their performance [5].

We pursued further analysis in this paper based on independent studies per resulting dimension during the COVID-19 era to highlight how these dimensions can be utilized to monitor and improve HCO's performance during the COVID-19 pandemic.

We believe that this paper has several strengths. First, this systematic review includes all types of studies with BSC implementations, such as books, theses, conference papers, and letters to the editor. Second, this review contains all implementations despite the country, language, or HCO administrative type, which gives an advantage of generalizing HCO results worldwide. Third, unlike other BSC reviews [35, 37], which included definitions of biobanks, pharmacies, laboratories, radiology, and medical colleges in HCO, this review limited the report to primary, secondary, or tertiary healthcare organizations. However, an initial assessment by top management to evaluate the importance of each dimension and KPI based on the health organizations' strategy could be needed. This strategy leads to the homogeneity of the resulting studies and leads to more valid comparisons among the results. Fourth, this review calculates the use frequency of perspectives and the weights/importance assigned to them. Fifth, the first review has uniform KPIs in homogenous major dimensions and subdimensions despite the categorization differences among implementations, yielding more precise results. The resulting KPIs and dimensions in this review can be generalized or replicable to other HCOs and hospitals. Finally, this study is the first to analyze the implications of BSCs in HCO during the pandemic based on the literature. This implication provides a guide for future theoretical implications, such as performing systematic reviews for each major dimension during the pandemic. It also provides a guide for practical implications of BSC dimensions to assess HCOs’ performance.

On the other hand, this systematic review has some limitations. First, unlike previous studies, it excludes some HCOs, such as laboratories, pharmacies, radiology departments, and biobanks, as specified in the inclusion/exclusion criteria. Therefore, our results cannot be generalized to such HCO types. However, we excluded them to arrive at more homogenous KPIs and dimensions that are directly related to HCO that offer primary, secondary, and tertiary medical services. Second, it includes only the articles that report the complete implementations of BSC while excluding studies that display only the BSC design without reporting the full implementation results. Third, we extracted the KPIs from all resulting implementations despite their RoB. However, we included an ROB assessment for each implementation.