Restoring a Sense of Touch: A Single Sensor Wearable Haptic Feedback System Based on Deep Learning and Arm Kinematics

Upper-limb prosthesis users usually lack comprehensive feedback or touch. This significantly hinders their ability to interact with their environment and have normal lives. This research involves the development and evaluation of a low-cost, non-invasive and easy to implement, wearable haptic feedback system. The system seeks to restore a sense of touch by interpreting arm movements through a wrist-worn 6-axis Inertial Measurement Unit (IMU). The setup is meant to be universal and to be used with most other upper-limb prosthetics to give sensory feedback to the user. Training and testing data were obtained from ten subjects performing a series of paired touch and non-touch activities. To perform real-time touch event classifications from the IMU data, various machine learning models were developed and evaluated. Multiple light-weight models were evaluated, including Logistic Regression, Support Vector Machines, Random Forest, and 1D-CNN. The best model, a 1D-CNN model, achieved 99.1% accuracy in classifying touch events of various different types. That model learned features directly from windowed time-series data and was trained using a combination of a partial Synthetic Minority Over-sampling Technique (SMOTE) and a Focal Loss function to prioritize the minority "touch" class. Model results improved when trained on initial contact instance rather than the full contact duration. These lightweight models were then successfully deployed on a Teensy 4.1 microcontroller as a proof of concept for the feasibility of using deep learning on IMU data for real-time sensory substitution.