Multimodal AI for Pilot Skill Assessment Using Physiological Signals
DOI:
https://doi.org/10.21590/ijtmh.12.02.06Keywords:
multimodal AI; pilot skill assessment; EEG; fNIRS; physiological signals; deep learning; mental workloadAbstract
Yet traditional methods of evaluating pilot skill, focused mainly on subjective instructor ratings or post hoc evaluation of aviation performance, only measure the observable products of cognitive functioning of skilled flying, and flight-related cognitive functioning is vitally important to modern aviation. This paper aims at developing four new frameworks: the MultiModal Aviator Performance and State Assessment AI Framework (MAPS-AI), that combines four physiological signals modalities (EEG, fNIRS, ECG and eye-tracking) and deep learning architectures for the purpose of real-time and objective monitoring of cognitive state in-flight while performing cognitive tasks during both flight training and flight operations. This framework brings together multi-sensor fusion and AI classification pipelines, limits of which are explored in the psychophysiological literature on pilot mental workload, the cognitive fatigue literature and EEG monitoring in actual flight conditions, fNIRS-based engagement detection in landing scenarios, multimodal workload assessment, and recent developments in compact EEG deep learning. A six-state cognitive state taxonomy is proposed and evidence is compiled demonstrating that incorporating progressive multimodal fusion yields a classification accuracy of around 83% compared a multimodal (EEG+EEG+EEG+EEG) approach with a accuracy of around 65% (EEG-only). Discussions of practical implications for aviation training, support systems for instructors, and human-machine interface are presented. The framework offers a conceptually sound basis for building an operationally viable assessment tools for pilots based on AI.
References
Barry, G., Johnstone, C., Caballero, W.N., Jenkins, P.R., Chou, C.A., Wang, Y. & Gaw, N. (2025). Student-pilot error prediction via multimodal physiological signals and tree-based models. IEEE Transactions on Cognitive and Developmental Systems. Advance online publication.
Causse, M., Dehais, F. & Pastor, J. (2011). Executive functions and pilot characteristics predict flight simulator performance in general aviation pilots. The International Journal of Aviation Psychology, 21(3), 217–234. https://doi.org/10.1080/10508414.2011.582441
Craik, A., He, Y. & Contreras-Vidal, J.L. (2019). Deep learning for electroencephalogram (EEG) classification tasks: A review. Journal of Neural Engineering, 16(3), Article 031001. https://doi.org/10.1088/1741-2552/ab0ab5
Dehais, F., Duprès, A., Blum, S., Drougard, N., Scannella, S., Roy, R.N. & Lotte, F. (2019). Monitoring pilot’s mental workload using ERPs and spectral power with a six-dry-electrode EEG system in real flight conditions. Sensors, 19(6), Article 1324. https://doi.org/10.3390/s19061324
Lawhern, V.J., Solon, A.J., Waytowich, N.R., Gordon, S.M., Hung, C.P. & Lance, B.J. (2018). EEGNet: A compact convolutional neural network for EEG-based brain-computer interfaces. Journal of Neural Engineering, 15(5), Article 056013. https://doi.org/10.1088/1741-2552/aace8c
Li, W., Li, R., Xie, X. & Chang, Y. (2022). Evaluating mental workload during multitasking in simulated flight. Brain and Behavior, 12(3), e2489. https://doi.org/10.1002/brb3.2489
Li, Y., Li, K., Wang, S., Chen, X. & Wen, D. (2022). Pilot behavior recognition based on multi-modality fusion technology using physiological characteristics. Biosensors, 12(6), Article 404. https://doi.org/10.3390/bios12060404
Rao, H., Cowen, E., Yuditskaya, S., Brattain, L., Koerner, J., Ciccarelli, G. & Heldt, T. (2022). Multimodal physiological monitoring during virtual reality piloting tasks. PhysioNet. https://physionet.org/content/multimodal-vr-pilot/1.0.0/
Schirrmeister, R.T., Springenberg, J.T., Fiederer, L.D.J., Glasstetter, M., Eggensperger, K., Tangermann, M., Hutter, F., Burgard, W. & Ball, T. (2017). Deep learning with convolutional neural networks for EEG decoding and visualization. Human Brain Mapping, 38(11), 5391–5420. https://doi.org/10.1002/hbm.23730
Verdière, K.J., Roy, R.N. & Dehais, F. (2018). Detecting pilot’s engagement using fNIRS connectivity features in an automated vs. manual landing scenario. Frontiers in Human Neuroscience, 12, Article 6. https://doi.org/10.3389/fnhum.2018.00006
Wilson, G.F. (2002). An analysis of mental workload in pilots during flight using multiple psychophysiological measures. The International Journal of Aviation Psychology, 12(1), 3–18. https://doi.org/10.1207/S15327108IJAP1201_2
Yao, J., Ma, J. & Dong, Y. (2025). A multimodal deep learning framework for pilot task performance prediction in within-visual-range air combat. IEEE Transactions on Aerospace and Electronic Systems. Advance online publication.
