Retrieval-augmented generation: a hybrid approach to assessing retrieved documents similarity, LLM confidence, and system stability

The retrieval-augmented generation approach relies on retrieving relevant documents to enhance large language model prompts and improve model outputs. However, existing metrics like cosine similarity, precision@k, and recall@k, to name just a few, fail to account for the confidence and stability of retrieval and generation. We propose a novel approach, retrieval confidence score, and its extension, asymptotic retrieval confidence score, which combines semantic similarity, large language model confidence, and stability across multiple generations. Asymptotic retrieval confidence score potentially provides a robust approach for evaluating retrieval-augmented generation systems, possibly suggesting a better solution for combining results across retrieval, generation, and evaluation stages.