Rethinking linguistic feedback: A modality-agnostic and holistic approach to multimodal addressee signals in spoken and signed dyadic interaction

In this paper, we investigate multimodal recipient feedback in casual dyadicconversation in four languages: German Sign Language, Russian Sign Language, spokenGerman, and spoken Russian. Taking a modality-agnostic and holistic approach,we investigate the composition of conversational feedback from different multimodalsignals, comparing sign and spoken languages without prioritizing any of the articulatorsor modalities. We find that in sign and spoken languages alike, feedback events includenon-manual signals such as head movements or facial expressions in 85% or more of theinstances. Moreover, we find that multimodal feedback signals cluster to form differentfeedback styles, ranging from a style employing a rich array of non-manual signals,over one comprising mostly head movements, to a style relying somewhat less on headand more on talk. Our data demonstrate that the basic infrastructure for feedback isshared among signers and speakers, while at the same time, signers and speakers showdifferent probabilities for using one style or another. Our study emphasizes the importanceof investigating interactional phenomena from a holistic, multi- and cross-modalperspective. As vocal and manual signals account only for a relatively small percentageof the feedback signals employed by the signers and speakers in our study, a linguistictheory that focuses solely on vocal and/or manual behavior remains incomplete andfails to account for the largest part of feedback in conversation. This study highlightsthat non-manual signals are fundamental to feedback and conversation more broadly,and argues that theories of language must be reconceptualized, as purely speech-basedaccounts fail to capture the full complexity of human interaction.