Beyond Signatures: Leveraging Sensor Fusion for Contextual Handwriting Recognition

This paper deals with biometric identification based on unique patterns and characteristics of an individual’s handwriting, focusing on the dynamic writing process on a touchscreen device. Related work in this domain indicates the dominance of specific research approaches. Namely, in most cases, only the signature is analyzed, verification methods are more prevalent than recognition methods, and the provided solutions are mainly based on using a particular device or specific sensor for collecting biometric data. In this context, our work aims to fill the identified research gap by introducing a new handwriting-based user recognition technique. The proposed approach implements the concept of sensor fusion and does not rely exclusively on signatures for recognition but also includes other forms of handwriting, such as short sentences, words, or individual letters. Additionally, two different ways of handwriting input, using a stylus and a finger, are introduced into the analysis. In order to collect data on the dynamics of handwriting and signing, a specially designed apparatus was used with various sensors integrated into common smart devices, along with additional external sensors and accessories. A total of 60 participants took part in a controlled experiment to form a handwriting biometrics dataset for further analysis. To classify participants’ handwriting, custom architecture CNN models were utilized for feature extraction and classification tasks. The obtained results showed that the proposed handwriting recognition system achieves accuracies of 0.982, 0.927, 0.884, and 0.661 for signatures, words, short sentences, and individual letters, respectively. We further investigated the main effects of the input modality and the train set’s size on the system’s accuracy. Finally, an ablation study was carried out to analyze the impact of individual sensors within the fusion-based setup.