A unified framework for reliability analysis in neuroimaging with Krippendorff’s α

This paper proposes using Krippendorff’s α for reliability analysis of neuroimaging data. Reliability analysis quantifies data robustness and is crucial for ensuring consistent results across different analysis pipelines or methods. Krippendorff’s α is a versatile statistic that works with nominal, ordinal, interval, and other data types, handling any number of raters and missing data. It measures the ratio between observed and expected agreement among raters using coincidence matrices.The paper explains how to calculate α and provides MATLAB code for implementation and examples of use on neuroimaging data. It includes a computationally efficient method for calculating α and a faster approximation method that maintains the logic of the exact test suited for large datasets usually found in neuroimaging. The uncertainty of the test statistic is calculated by bootstrapping. This work aims to simplify reliability analysis in neuroimaging, making it accessible for researchers.