Langmark: annotations for scenes with inconsistent objects connecting distributional semantic models to vision science

We introduce Langmark: object annotations (outlines and labels) for 124 scenes containing different violations of environmental regularities, such as semantic inconsistencies, i.e., objects that are unexpected in a given context, like a hammer in a kitchen. Langmark enables the analysis of these scenes, which are popular stimuli in scene-perception research, using various methods, including distributional semantic models (DSMs) – tools that encapsulate human linguistic knowledge. To showcase the potential of this approach, we demonstrate that an off-the-shelf DSM reliably distinguishes inconsistent objects from their consistent counterparts and therefore captures human intuitions about different objects being consistent with a given scene or not. Moreover, the model expresses object-scene consistency on a continuous scale, which facilitates testing hypotheses rooted in the assumption that humans also treat it as gradual. As a step in this direction, we compared model-derived consistency scores to human ratings of object-scene consistency collected in several ways. We discovered that these ratings depend on several factors related to experimental procedure and design. Based on our findings, we created practical guidelines for collecting such ratings. These guidelines and the Langmark annotations should, respectively, inform the research on scene perception and enable a wider adoption of DSMs within vision science. The annotations are available here: https://zenodo.org/records/13341543.