Can subjective agreement levels be measured on an interval scale? An empirical test of individual-level response structures

Numeric ratings of agreement (also known as Likert-scales) are one of the most common assessment tools in psychology. However, little is known about the measurement theoretic properties of such ratings. Most prominently, the question whether such ratings yield interval scaled measurements has been questioned on theoretical grounds, the key issue being that interval scales do not only require ratings to be order preserving, but also additive. In this article, I apply a representational measurement model to investigate intra-individual response structures with regard to Likert-scale items. The model is borrowed from the psychophysical paradigm of cross-modality comparisons and allows for an explicit empirical test of additivity on the level of individual responses. A within-subjects experiment with N = 140 individuals was conducted, where subjects were instructed to rate their agreement to a set of items from a standardized personality questionnaire. The ratings were performed repeatedly with varying reference stimuli to evoke difference ratings for agreement. These difference ratings were then used to test the conditions of the measurement model. The results indicate that 51% of the individuals showed additivity in their rating behavior. Therefore, an interval scale representation of subjective agreement was possible for about one half of the individuals, whereas the other half did not produce agreement ratings on an interval scale. While the results clearly demonstrate that interval scale measurements using numeric ratings are possible, they also indicate that such quantitative representations should not be taken at face value but need to be tested empirically.