Computational Modeling of Thought-Feeling Accuracy Using Sentence Embeddings

Thought-feeling accuracy, the ability to accurately infer another person's dynamic internal states during social interactions, is a fundamental social cognitive skill. However, research on this construct has advanced slowly due to the significant time and labor required for its measurement, which involves collecting written reports of thoughts and feelings and training independent human coders to rate the accuracy of perceivers' inferences. The present research explores whether Natural Language Processing (NLP) embedding models can provide a scalable, efficient, and replicable alternative to human coding for rating thought-feeling accuracy.Embedding models convert text—in this case, targets' reported thoughts/feelings and perceivers' inferences—into numerical vectors in a multidimensional space, allowing semantic similarity (a proxy for accuracy) to be quantified mathematically. We employed a comprehensive set of 14 pre-trained NLP models (bi-encoders and cross-encoders), including proprietary state-of-the-art systems, and two models fine-tuned specifically for the task. We analyzed a large dataset of over 29,000 thought-inference pairs collected across 10 past studies. Comparing human and model ratings, the aims of this study were (1) quantifying the agreement, (2) measuring the disagreement, (3) assessing convergent validity, (4) exploring potential sources of disagreement such as linguistic and sample characteristics, and (5) evaluating the gains from fine-tuning models on the thought-feeling accuracy data. Preliminary results show promising strong associations between model and human ratings, suggesting that the human-coded construct is, at least in part, a linguistic phenomenon that can be operationalized through semantic similarity.However, differences also emerge, especially at lower levels of accuracy, highlighting nuances that human coders may capture (e.g., context, emotional tone) that text-based models may miss. If successful, this computational approach could significantly reduce measurement bottlenecks, enable larger and more diverse studies, and provide new tools for interpreting the psychological meaning of linguistic content in social cognition research.