A Unified Framework for Reliability Analysis in Neuroimaging With Krippendorff's α

This paper proposes Krippendorff's α for reliability analysis of neuroimaging data. Reliability analysis quantifies the robustness of data and is crucial for ensuring consistent results across different analysis pipelines or methods. It measures the ratio between observed and expected agreement among raters using coincidence matrices. The paper explains how to calculate α and provides MATLAB code for implementation, along with examples of use on neuroimaging data. It includes a computationally efficient method for calculating α and a faster approximation method that maintains the logic of the exact test, making it suitable for large datasets typically found in neuroimaging. The uncertainty of the test statistic is estimated by bootstrapping. This work aims to simplify reliability analysis in neuroimaging, making it accessible for researchers.