Eyes on the Prize: A Frame-Semantic Gaze at Referents of Action
Listed in
This article is not in any list yet, why not save it to one of your lists.Abstract
This study investigates within the framework of Frame Semantics and building on Event Segmentation Theory how gaze patterns align with semantic structures in dynamic visual contexts, focusing on the referents of action. Using the Framed Multi30K dataset—a multimodal dataset enriched with frame semantic annotations—an eye-tracking experiment was conducted to analyze participants’ visual attention under two conditions: non-verbal tasks and English audio descriptions. Stimuli consisted of images depicting dynamic Cause_Motion events annotated with semantic frames to examine how attentional allocation corresponds to event-relevant frame elements, such as agents, themes, and paths. This study demonstrates a proof-of-concept pipeline to enable systematic, hypothesis-testing investigations of event-driven attention and, more broadly, event cognition, by aligning and integrating fixation data with annotations, with potential implications for multimodal applications, such as imagecaptioning and grounded language understanding.
