Seeing a talker’s mouth reduces the effort of perceiving speech and repairing perceptual mistakes for listeners with cochlear implants

Objectives: Seeing a talker’s mouth improves speech intelligibility, particularly for listeners who use cochlear implants (CIs). However, the impacts of visual cues on listening effort for listeners with CIs remain poorly understood, as previous studies have focused on listeners with typical hearing (TH) and featured stimuli that do not invoke effortful cognitive speech perception challenges. This study directly compared the effort of perceiving audiovisual speech between listeners who use CIs and those with TH. Visual cues were hypothesized to yield more relief from listening effort in a cognitively challenging speech perception condition that required listeners to mentally repair a missing word in the auditory stimulus. Eye gaze was simultaneously measured to examine whether the tendency to look toward a talker’s mouth would increase during these moments of uncertainty about the speech stimulus. Design: Participants included listeners with CIs and an age-matched group of participants with typical age-adjusted hearing (N = 20 in both groups). The magnitude and time course of listening effort was evaluated using pupillometry. In half of the blocks, phonetic visual cues were removed by selectively blurring the talker’s mouth, which preserved stimulus luminance so visual conditions could be compared using pupillometry. Each block included a mixture of trials in which the sentence audio was intact, and trials in which a target word in the auditory stimulus was replaced by noise; the latter required participants to mentally reconstruct the target word upon repeating the sentence. Pupil and gaze data were analyzed using generalized additive mixed effects models (GAMMs) to identify the stretches of time during which effort or gaze strategy differed between conditions.Results: Visual release from effort was greater and lasted longer for listeners with CIs compared to those with TH. Within the CI group, visual cues reduced effort to a greater extent when a missing word needed to be repaired than when the speech was intact. Seeing the talker’s mouth also improved speech intelligibility for listeners with CIs, including reducing the number of incoherent verbal responses when repair was required. The two hearing groups deployed different gaze strategies when perceiving audiovisual speech. CI listeners looked more at the mouth overall, even when it was blurred, while TH listeners tended to increase looks to the mouth in the moment following a missing word in the auditory stimulus. Conclusions: Integrating visual cues from a talker’s mouth not only improves speech intelligibility, but also reduces listening effort, particularly for listeners with CIs. For listeners with CIs (but not those with TH), these visual benefits are magnified when a missed word needs to be mentally corrected — a common occurrence during everyday speech perception for individuals with hearing loss. These results underscore the importance of including participants with hearing loss in listening effort studies and suggest caution in assuming results from TH listeners will generalize to those with hearing loss. They also highlight the potential clinical relevance of visual speech information, for counseling patients and families and potentially for the development of audiovisual strategies to reduce listening effort.