The impact of factor score extraction method for exploratory factor analysis on resultant factor scores and links to behaviour in a cognitive task across independent samples differing in underlying correlation structure

For factor analysis, the link between factors and factor scores (i.e. the score each person has for a factor) is not unique. Therefore, there are different methods that have been proposed for extracting factor scores. In computational psychiatry, researchers commonly use designs of two separate samples (discovery and replication sample) in which they train a factor analysis in one sample and then apply it to extract factor scores to the second sample, extracting factor scores. Here, I compared the impact of different factor extraction methods for questionnaire data on the correlation structure between the extracted factor scores and on out-of-sample predictions of the questionnaire factors based on performance in a cognitive task. For this, I re-analyzed our openly available data (Trier et al., 2025) using the ‘psych’ package (Revelle and Revelle, 2015) in R. I find across several measures that the Bartlett method is mildly preferable to the default method in the psych package (Thurstone). Specifically with the Bartlett method, the correlations between factor scores more closely reflect the correlation structure in the raw data, the factor scores are more similar to the ones used traditionally when scoring questionnaires (‘non-weighted sum’) and the factor scores of each person only depend on the questionnaire responses of that person and not on the questionnaire responses of other participants. Additionally, links between task behaviour and factor scores is mildly stronger for factor scores extracted the Bartlett method. In conclusion, using the ‘Bartlett’ method has some advantages when extracting factor scores on a new sample.