Development of Multimodal Augmented Reality Application on the Flight Deck for Single-Pilot Operations: System Usability and User Interface Satisfaction

Implementing augmented reality (AR) technologies has become a popular method of increasing operators' situation awareness by adding virtual information to the physical environment. In the current commercial two-pilot flight deck, the pilot-flying (PF) is responsible for flying the aircraft to an approved flight plan, and the pilot-monitoring (PM) focuses on communicating and monitoring PF’s operational behaviours. The driving factors behind single-pilot operations (SPO) are the foreseen pilot shortage and desire to reduce operating costs. Whilst SPO is expected to be enabled - in part - by advanced flight deck technologies. Forty participants attended simulator trials that involved interacting with multimodal AR apps (Hololens) which included voice and gesture command functionalities. Results revealed voice command scored higher than the gesture command on both the System Usability Scale (SUS) and the Questionnaire for User Interface Satisfaction (QUIS). AR visualisation that blends the physical operation environment and a virtual holographic checklist with guiding cues can improve pilots’ monitoring performance and procedure compliance during the instrument landing trials. Furthermore, the AR voice command permits multiple sensory processing and response by integrating visual, auditory, and tactile inputs simultaneously, which provides the pilot with greater flexibility to meet task requirements. AR gesture command was regarded as an unnecessary burden to the pilot’s cognitive resources and limited hand movement while executing complicated operating procedures. Future research shall explore the implementation of AR voice command with artificial intelligence (AI) on the flight deck to support a single pilot performing both flying and monitoring tasks. This research paradigm needs to be assessed thoroughly for human-centred design to ensure broad acceptance by end users, manufacturers, airlines and regulators.