Segment Anything Model for Industrial Vision: A Comprehensive Evaluation with a New Metric, a New Dataset, and a Toolbox

Image segmentation is a fundamental task of computer vision, and plays a vital role in intelligent manufacturing for detection of surface defects. Recent emergence of segmentation foundation models, such as segment anything model (SAM), provide remarkable versatility and performance across various segmentation tasks with natural images. For industrial vision, defect image segmentation poses a significant challenge due to limited sample sizes, vast scene variations, and diverse defect shapes, etc. There are three issues on evaluating SAM's performance on industrial images: existing public defect datasets are too simple and small, and the performance metrics tend to saturate; are the existing metrics objective for such special defect images? And how to evaluate SAM easily? Therefore, this study aims to comprehensively evaluate SAM's performance on industrial images. Firstly, we release a large dataset of mobile phone screen for enhancing data diversity. Secondly, a new evaluation metric is proposed for fitting in the industrial defects. And lastly, an open toolkit is available for easily transferred to other defect images. We also identify the potential paths for future research, and believe that our contribution is beneficial for the implementation of vision foundation models, such as SAM, in both academic and industrial communities.