Interacting with LLM-powered virtual humans in AR and VR: Does medium mediate social presence and realism?

Interactive virtual humans are increasingly embedded in XR experiences across domains, from leisure, education, and training to healthcare. To be effective, these virtual humans must evoke a strong sense of social presence and realism, appearing believable, socially responsive, and co-present. Recent advances in artificial intelligence have made it easier to deploy agents that can naturally converse with users, enabling more lifelike interactions. While both Virtual Reality (VR) and Augmented Reality (AR) can host these virtual humans, little is known about how the virtual humans' display medium influences users’ perceptions and interaction. Here, we investigated whether the interaction medium (AR vs VR) with an an LLM-powered virtual human affects social presence and markers of social realism, like interpersonal distance and heart rate variability (HRV) of the user. Our results show that the display medium influences the users' perception of social presence, interpersonal distance, and the HRV. Participants kept a greater interpersonal distance from the virtual human when interacting with the it in AR compared to VR. Interactive virtual humans also induced a change in heart rate variability, indicating higher arousal levels when conversing with the interactive virtual human. Furthermore, we found participants liked the virtual human more when it was displayed in AR than in VR. Our results highlight the importance of selecting display medium when designing experiences with interactive virtual humans, as AR and VR elicit distinct paterns of social presence, interpersonal behavior, and physiological arousal.