Comparing 2D SLAM Algorithms: A Novel Benchmarking Framework for Map Quality Assessment

Evaluating SLAM algorithms is critical in determining their performance and suitability for specific applications. This paper introduces a novel benchmark framework for quantitatively assessing 2D SLAM map quality through two stages: preprocessing and evaluation. The preprocessing stage focuses on correcting the SLAM map according to the ground truth map to ensure comparability. This consists of image registration and thinning operation. In the evaluation stage, the resulting SLAM map is compared with the ground truth using various methods: Map Similarity, Geometric Distance Measurement, and Correspondence Matching. To aid in environment selection, we introduce the Environment Complexity Matrix (EnviCM), which categorizes environments into simple, complicated, and complex based on coverage area and feature density. EnviCM assists researchers and developers in making informed decisions regarding SLAM algorithm suitability for the chosen environment, leveraging outcomes from the evaluation stage. The evaluated SLAM algorithms are the widely used Gmapping, Hector SLAM, and Cartographer.