Beyond Accuracy: A Critical Review of Document Image Binarization Evaluation Practices

Document image binarization is a fundamental yet challenging task in document analysis, particularly for historical documents affected by complex degradations such as uneven illumination, bleed-through, stains, and ink fading. While substantial progress has been made, algorithms performance strongly depends on document characteristics and acquisition conditions. To support objective comparison, benchmarking initiatives such as the Document Image Binarization Competition (DIBCO) provide standardized datasets and pixel-level evaluation protocols. While existing studies largely focus on binarization accuracy, practical deployment additionally requires careful consideration of computational efficiency, as high-quality methods may incur prohibitive processing costs. This paper presents a critical review of document image binarization evaluation methodologies, with a particular focus on historical documents. We analyze the evolution of algorithms evaluated within the DIBCO frameworks, examine the strengths and limitations of commonly used performance metrics, and discuss the impact of blind evaluation protocols on algorithm ranking. Furthermore, we highlight the emerging importance of time-quality trade-offs and efficiency-aware evaluation criteria. The survey highlights the absence of a universally optimal binarization method and points to the need for adaptive, degradation-aware strategies and more comprehensive evaluation practices in future research.