Visual attention-based pilots’ status monitoring and performance enhancement towards human-in-the-loop automation

Abstract

Automation systems have increasingly been implemented in the aircraft's cockpits to facilitate human pilots and reduce their workload. While automation systems have significantly improved operational efficiency and reduced human errors, poorly designed human-computer interaction (HCI) in cockpits has also introduced new challenges in the highly automated control loops, such as the impaired cognitive status induced by Out-Of-The-Loop (OOTL) phenomenon and the overload of the In-The-Loop (ITL) pilots caused by the multiple information resources in the cockpits. The existing strategies mostly focus on solely identifying the pilot's OOTL status with machine learning methods or providing assistance with predefined protocols based on the pilot's workload levels. However, these methods lack explainability and fail to offer proactive automation support specifically tailored to pilots' cognitive states and task demands. Therefore, closed-loop support is required to provide human-centric assistance to both the OOTL and ITL pilots with user-friendly interactions. This thesis presents a systematic approach leveraging eye-tracking technology and artificial intelligence (AI) to enhance closed-loop support for pilots. This work addresses three key research problems, with contributions as follows. First, a novel Flashlight model integrates attention distribution and attention resource metrics, providing a comprehensive framework to analyze pilots' visual attention and predict operational performance. Second, the Visual Attention LTLf for Identifying OOTL (VALIO) framework employs linear temporal logic and graph neural networks to identify OOTL status with enhanced explainability, offering human-readable insights into pilots' behaviours. Third, an innovative Large Language Model (LLM)-based method tokenizes eye-tracking data into Visual Attention Matrices (VAMs) to detect and support ITL troubleshooting behaviours, enabling context-aware and resource-efficient human-computer interactions. Several case studies were conducted at the Human Factors Lab in the Department of Aeronautical and Aviation Engineering (AAE) to verify the efficacy of the proposed methods. The eye-tracking measurements developed based on the Flashlight model improved the prediction accuracy of pilots' operation performance. The VALIO framework achieved a stable identification accuracy across different time windows, with F1 scores around 0.8. And the explainability is significantly increased by the generated human-readable formulas. The integration of eye-tracking techniques and LLM achieved a Micro-average F1 score of 0.852 for identifying where the pilot is troubleshooting, with proactive and user-friendly interactions. In conclusion, this thesis contributes to aviation safety by developing innovative methods for monitoring, predicting, and supporting pilot performance in both OOTL and ITL scenarios, advancing the human-in-the-loop HCI in modern cockpits. These developments lay the groundwork for safer and more efficient aviation operations.