Video-mediated communication modulates conversational features of speech more than gesture in novel referential communication

For face-to-face communication to flow smoothly, people engage in various activities, such as taking turns rapidly, using hand gestures, and aligning in word choices and gestures, to name a few. Nowadays, many conversations occur on videoconferencing platforms, making them important settings for social interactions. However, it is underexplored how these multimodal conversational features compare across modes of communication. By analyzing two corpora of multimodal interactions—one for co-present face-to-face interaction and the other for video-mediated interaction—we found strikingly similar patterns of speech and gesture production across modes of communication. In particular, we found a great amount of overlap in the distributions of floor transition offset (FTO), turn-length, the frequency and size of co-speech gestures, and the frequencies of lexical and gestural alignment, across conversational rounds. However, there were also differences: Online video-mediated communication was associated with (i) longer floor transfer offsets (454 ms) and fewer overlaps, (ii) a greater reduction in the number of words over time, (iii) a lower lexical alignment rate at the beginning of the conversation, and (iv) a greater proportion of alignment of iconic gestures with similar forms than face to face communication. These findings suggest that the fundamental dynamics of conversation are nearly identical between co-present face-to-face and video-mediated interactions, but video-mediated communication modulates some aspects of speech-related interactive language use more than those of co-speech gestures.