Perceptual learning of speaker-specific gesture-speech temporal alignment: Effects on word recognition

Perceptual learning helps listeners cope with the lack of invariance in speech perception. Exposure to a nontypical phoneme in disambiguating contexts can bias subsequent perception. Such learning is driven not only by lexical context but also by visual cues. One such visual cue is the beat gesture, a simple up-and-down hand movement that usually co-occurs with stressed syllables in everyday conversation. Beat gestures can influence stress perception and thus bias word recognition. However, gesture-speech temporal alignment varies substantially across speakers. The current study examined whether listeners can learn about speaker-specific gesture-speech alignment patterns from disambiguating auditory stress cues. Across three experiments, participants were exposed to a speaker consistently gesturing earlier than stressed syllables in one group, and later in the other. After exposure, both groups categorized minimal stress pairs (e.g., VOORnaam–voorNAAM) with ambiguous auditory stress cues and a beat midway between two target syllables. A reliable group difference emerged when the test stimuli were sufficiently ambiguous: word recognition was biased by the gesture-speech alignment learnt during exposure. However, this learning appeared not to generalize to novel words. Our results demonstrate that the learnt gesture-speech alignment can be applied to word recognition for the same speaker, optimizing face-to-face interaction.