Sensory Substitution and Embodiment: Shaping Performance and Immersion in Teleoperated Settings

Background: Telemanipulation enables the execution of remote operations using specialized tools, often under reduced sensory input, such as lack of haptic feedback. With practice, these tools can become extensions of the user's body, a phenomenon known as embodiment, thereby enhancing telepresence—the perception of being present at the action site. Sensory substitution (SS) can mitigate sensory limitations in such tasks.Objectives: To study the learning process of sensory substitution, taking into account the levels of embodiment and telepresence experienced during the task, we developed the Visual-Tactile Sensory Substitution Task (VTSST).Methods: Forty participants manipulated 200 virtual elastic slabs of varying elasticity using a haptic stylus, aiming to avoid breakage. Half the trials included force feedback (control condition), while the other half relied on visual cues (SS condition). Performance metrics included Elapsed Time (ET) to deform each slab, Maximal Applied Force (MAF), and Frequency of Errors (FoE), along with pupil dilation and blink rate as cognitive workload indicators. Participants also assessed their perceived SS, embodiment, and telepresence.Results and Conclusions: Results showed the SS condition led to lower ET, higher MAF, and more FoE than control trials, but ET and F changed over time in the SS condition, indicating learning. Blink rate, reflecting workload, was lower in SS trials, especially in the late phase of the task, while pupil dilation remained unchanged.Structural Equation Modeling revealed that perceived SS positively influenced embodiment, presence, and SS learning. In addition, embodiment enhanced presence, which reduced workload. This study highlights how SS supports telemanipulation by fostering embodiment, reducing task time, and improving multisensory integration for efficient remote operation