Does modality incongruency impair veracity memory? Asymmetrical effects on false-labeled information

Mental representations are mostly not unimodal but multimodal, allowing information acquired in one modality to be retrieved in another. This research investigates how modality (in)congruency between encoding and retrieval influences veracity memory of information labeled as true or false. In three experiments (N = 1.096), participants first encoded event descriptions stepwise as text (Experiment 1 & 2) or as graphics (Experiment 3) before retrieving them in a congruent (same as encoding: text in Exp. 1 and 2; graphic in Exp. 3) or incongruent modality (different from encoding: graphic in Exp. 1 and 2; text in Exp. 3). Results from Experiments 1 and 2 showed that modality incongruency (Text-to-Graphic) reduced veracity memory, particularly for false-labeled information, supporting a modality-veracity interaction. Experiment 3 found no significant difference in accuracy between congruent and incongruent retrieval when participants encoded information in graphics (Graphic-to-Graphic vs. Graphic-to-Text). However, the veracity of false-labeled information was less often accurately recalled than the veracity of true-labeled information. These findings indicate that retrieving veracity information across different modalities is asymmetrical. While Text-to-Graphic transfer impaired veracity memory for false-labeled information, Graphic-to-Text transfer had no such interaction effect. This asymmetry suggests that the initial encoding modality influences how vulnerable veracity memory is to disruption when retrieval happens in a different modality.