A systematic review: the dimensions to evaluate health care performance and an implication during the pandemic

  1. Faten Amer
  2. Sahar Hammoud
  3. Haitham Khatatbeh
  4. Szimonetta Lohner
  5. Imre Boncz
  6. Dóra Endrei

Reviewed by

Read the full article

Listed in

Log in to save this article

Abstract

Background

The balanced scorecard (BSC) has been implemented to evaluate the performance of health care organizations (HCOs). BSC proved to be effective in improving financial performance and patient satisfaction.

Aim

This systematic review aims to identify all the perspectives, dimensions, and KPIs that are vital and most frequently used by health care managers in BSC implementations.

Methods

This systematic review adheres to PRISMA guidelines. The PubMed, Embase, Cochrane, and Google Scholar databases and Google search engine were inspected to find all implementations of BSC at HCO. The risk of bias was assessed using the nonrandomized intervention studies (ROBINS-I) tool to evaluate the quality of observational and quasi-experimental studies and the Cochrane (RoB 2) tool for randomized controlled trials (RCTs).

Results

There were 33 eligible studies, of which we identified 36 BSC implementations. The categorization and regrouping of the 797 KPIs resulted in 45 subdimensions. The reassembly of these subdimensions resulted in 13 major dimensions: financial, efficiency and effectiveness, availability and quality of supplies and services, managerial tasks, health care workers' (HCWs) scientific development error-free and safety, time, HCW-centeredness, patient-centeredness, technology, and information systems, community care and reputation, HCO building, and communication. On the other hand, this review detected that BSC design modification to include external and managerial perspectives was necessary for many BSC implementations.

Conclusion

This review solves the KPI categorization dilemma. It also guides researchers and health care managers in choosing dimensions for future BSC implementations and performance evaluations in general. Consequently, dimension uniformity will improve the data sharing and comparability among studies. Additionally, despite the pandemic negatively influencing many dimensions, the researchers observed a lack of comprehensive HCO performance evaluations. In the same vein, although some resulting dimensions were assessed separately during the pandemic, other dimensions still lack investigation. Last, BSC dimensions may play an essential role in tackling the COVID-19 pandemic. However, further research is required to investigate the BSC implementation effect in mitigating the pandemic consequences on HCO.

Article activity feed

  1. Version published to 10.1186/s12913-022-07863-0
  2. Version published to 10.1101/2021.06.26.21259568v2 on medRxiv
  3. ScreenIT

    SciScore for 10.1101/2021.06.26.21259568: (What is this?)

    Please note, not all rigor criteria are appropriate for all manuscripts.

    Table 1: Rigor

    Ethicsnot detected.
    Sex as a biological variablenot detected.
    Randomizationnot detected.
    Blindingnot detected.
    Power Analysisnot detected.

    Table 2: Resources

    Software and Algorithms
    SentencesResources
    Data Sources, Search Strategy, and Study Selection: First, a search strategy was developed for the PubMed database, see Table (1).
    PubMed
    suggested: (PubMed, RRID:SCR_004846)
    Then, the strategy was adapted for Embase, Cochrane CENTRAL, and Google Scholar databases.
    Embase
    suggested: (EMBASE, RRID:SCR_001650)
    Cochrane CENTRAL
    suggested: (Cochrane Central Register of Controlled Trials, RRID:SCR_006576)
    Google Scholar
    suggested: (Google Scholar, RRID:SCR_008878)
    To widen the search frame and get more results, the search strategy was developed to include both Medical Subject Headings (MeSH) terms and keywords.
    MeSH
    suggested: (MeSH, RRID:SCR_004750)
    Afterward, the 1st author conducted an electronic database search and removed the duplicates by using the EndNote X9.2 program.
    EndNote
    suggested: (EndNote, RRID:SCR_014001)
    For that, the frequency of use for each KPI at each implementation was plotted on Microsoft Excel and the sum was calculated.
    Microsoft Excel
    suggested: (Microsoft Excel, RRID:SCR_016137)
    Each Risk of Bias (RoB) was categorized into 3 categories according to the Cochrane Handbook for Systematic Reviews: high risk, low risk, and unclear risk [46].
    Cochrane Handbook
    suggested: None
    Summary of RoB was prepared using RevMan 5 [47].
    RevMan
    suggested: (RevMan, RRID:SCR_003581)

    Results from OddPub: We did not detect open data. We also did not detect open code. Researchers are encouraged to share open data when possible (see Nature blog).

    Results from LimitationRecognizer: We detected the following sentences addressing limitations in the study:
    Moreover, researchers are encouraged to perform systematic reviews for each dimension, especially those dimensions that are already well investigated, as well as the investigation of dimensions that are still poorly investigated but essential for PE. 3.10 Strengths and weaknesses: We believe that this paper has several strengths. First, this systematic review includes all types of studies with BSC implementations, such as books, theses, conference papers, letters to the editor, etc. Second, this review includes all implementations despite the country, language, or HCO administrative type. This gives an advantage of being able to generalize the results on HCOs worldwide. Third, this review not only calculates the frequency of use of perspectives but also calculates the weights (importance) assigned to them. Fourth, it is the first review that has uniformed the KPIs in homogenous major dimensions and sub-dimensions despite the categorization differences among implementations, which yields more precise results. Finally, this study is the first to analyze the implication of BSC in HCOs during the pandemic. The resulted KPIs and dimensions at this review can be generalized or replicable to other HCOs and hospitals. However, an initial assessment needs to be done by top management to evaluate the importance of each according to the health organizations’ strategy. On the other hand, this systematic review has some limitations. First, unlike previous reviews, it excludes some HCOs su...

    Results from TrialIdentifier: No clinical trial numbers were referenced.

    Results from Barzooka: We did not find any issues relating to the usage of bar graphs.

    Results from JetFighter: We did not find any issues relating to colormaps.

    Results from rtransparent:
    • Thank you for including a conflict of interest statement. Authors are encouraged to include this statement when submitting to a journal.
    • Thank you for including a funding statement. Authors are encouraged to include this statement when submitting to a journal.
    • No protocol registration statement was detected.

    Results from scite Reference Check: We found no unreliable references.

    About SciScore

    SciScore is an automated tool that is designed to assist expert reviewers by finding and presenting formulaic information scattered throughout a paper in a standard, easy to digest format. SciScore checks for the presence and correctness of RRIDs (research resource identifiers), and for rigor criteria such as sex and investigator blinding. For details on the theoretical underpinning of rigor criteria and the tools shown here, including references cited, please follow this link.

  4. Version published to 10.1101/2021.06.26.21259568v1 on medRxiv