Assessing complex belief structures with the triads task: Efficiency of task design and levels of analysis

The triads task is a versatile method that allows for assessing beliefs as networks ofconcepts through their perceived similarity. In the task, participants make judgements about thesimilarity of predefined concepts of interest which are presented in triads. Based on thosejudgements, a similarity score can be calculated for each pair of concepts. Adding more conceptsto a triads task design quickly leads to a combinatorial explosion of trials, which can make thetriads task inefficient. In our study, we explore two ways of shortening the triads task: leaving outconcepts from the pool and balanced incomplete block designs. We employed a triads task withconcepts from the topical domains of Open Science and digitization and examined how wellsimilarity scores calculated from the complete triads task design and from reduced designs werecorrelated as a marker of correspondence. Reducing the underlying concept pool and thereby thenumber of trials in the triads task led to highly correlated similarity scores but is less efficient inreducing the number of trials than balanced incomplete block designs. Similarity scores from thebalanced incomplete block designs were highly correlated with those from the complete designsand higher compared to the correlations of similarity scores from split-half designs and thecomplete designs. We additionally explored questions of employing rotated balanced incompleteblock designs and of sample homogeneity. Our results suggest that both methods are in their wayappropriate for making the triads task more efficient. We discuss the respective trade-offs withregards to loss of information.