Using Natural Language and Health Ontologies in Hope Recommender System: Evaluation of Use in Primary Care

Objective To evaluate the accuracy of the results of an evidence-based information recommender called Health Operation for Personalized Evidence (HOPE), which is a UMLS ontology-based solution and was used as an automatic assistant to find information in primary care cases. Materials and methods Ontologies and Natural Language Processing (NLP) were used by HOPE to assess clinical cases and generate a group of recommendations. The results were reviewed by fifty general practitioners (raters). The Kappa-Fleiss Coefficient measured the level of agreement among them. The precision was also measured to evaluate the satisfaction of a user who is presented with different EBM references. The precision (precision@k) and Normalized Discounted Cumulative Gain (NDCG@k) were also calculated because both metrics help assess different aspects of AI recommendation systems. Results The Kappa Fleiss coefficient for the 50 raters was 0,66 (z=277 and p-value=0) which represents an “important association” among raters. The results for precision@3, precision@4 and precision@5 over 0,75 showed also good results (@3 72%; @4 72%; @5 68%) as well as the NDCG@k (NDCG@3 63%; NDCG@4 60%; NDCG@5 60%). The results presented after the NDCG@k indicate that the recommendation system adequately classifies the recommendations presented by HOPE according to the ratings of the raters. The results are highly relevant and they are in the optimal order according to the preferences of raters. HOPE’s response service time seems to be almost immediate, with a mean time of 17,4 seconds. Conclusion The recommender system seems to find accurate information for primary care. The use of UMLS ontologies and NLP allows HOPE to find useful information for a clinical case within a reasonable time (relative to the patient’s time assigned). These recommendations could potentially get better diagnosis and treatment as well as eventually reduce consultation time.