A multi-sensory experience of touch for TechnoSapiens

The TechnoSapiens system was developed as a platform for conducting psychological experiments on the self- and other-perceptions of users of bionic technologies in AR (Rudolph et al. (2024)). Using diminished reality (DR) techniques, this system visually replaces the user’s arm with a bionic prosthesis model that is intuitively controlled by hand and arm movements. However, due to the lack of haptic feedback, users experienced difficulties in interacting with virtual objects. In the present work, we report on the extension of TechnoSapiens to provide a multi-sensory interaction experience that incorporates visual and haptic feedback. To this end, we have developed a novel data glove that provides haptic feedback and meets the requirements of the DR system. In addition, we use a physics simulation to prevent the prosthesis from penetrating virtual objects during interaction, i.e., together with the haptic feedback, we provide appropriate visual feedback of touching an object. We conduct a user study showing that this multi-sensory feedback improves the sense of body ownership, immersion, and self-presence. We provide source code for the data glove micro-controller and the Unity3D integration.