Adaptation of eye movement behaviour during closed-captioned movie viewing

Closed captions are increasingly present in movie viewing, exceeding their initial purpose for hearing-impaired audiences to a wider use among viewers. Integrating captions introduces an additional visual element to the multisensory experience of watching movies. This study examines how eye movement behaviour differs between free reading of closed captions and background scene gazing and how it is affected by the availability of audio. Using eye-tracking, we analysed saccades and fixations in designated areas of interest as participants viewed a 20-minute movie with the alternating presence and absence of audio. Viewers seamlessly integrated the two visual input streams by frequently shifting their gaze between the captions and the scene, even in the presence of audio. In the absence of audio, viewers adjust their transition probabilities between reading captions and observing visual scenes - rather than the duration of reading intervals - altering the balance to adapt to the missing auditory input. Modelling the characteristics of eye movements with gamma functions revealed less skewed distributions on the caption text compared to the visual scene, indicating more consistent eye movement patterns on caption text than the background natural scenes. Notably, these patterns remained stable regardless of the presence or absence of sound. These findings highlight how reading and gaze-related eye movement behaviours are shaped by perceptual features, with viewers adapting to auditory absence by increasing gaze shifts without fundamentally changing low level eye movement properties. Language, emotional content, and narrative elements strongly influence caption engagement, underscoring the role of content in directing attention within multisensory media experiences.