目的 情境意识（Situation Awareness, SA）作为反映驾驶员对列车运行环境认知能力的高阶心理变量,较传统的单一心理状态指标更能全面表征驾驶安全状态。为突破现有铁路司机心理状态监测手段的局限性,提升智能铁路行车安全体系的水平。本文基于眼动、心电和脑电3类生理数据,开展SA状态的多模态辨识研究。方法 针对SA状态难以动态获取、精确标注及可解释性不足的问题,提出覆盖SA诱发、标注、建模与解释的系统性研究方法,揭示多模态生理特征在SA辨识中的作用机制。针对生理信号非平稳性导致模型泛化能力受限的问题,构建混合特征选择策略的领域泛化方法,提升了模型在跨个体与跨会话条件下的适应性。此外,针对脑电（Electroencephalogram, EEG）信号采集受限问题,构建基于潜在空间建模与条件生成机制的多模态学习方法,实现了在EEG模态缺失条件下的鲁棒SA辨识。结论 为构建高可靠性、高适应性的列车驾驶安全状态监测系统提供了理论与方法支持,提升列车驾驶人机协同系统的智能化与安全性。
Abstract
Situation Awareness (SA), as a higher-order psychological variable reflecting train drivers' cognitive ability of the operating environment, provides a more comprehensive characterization of driving safety status compared with traditional single psychological state indicators. The work aims to conduct multi-modal recognition research on SA states based on three types of physiological data: eye movement, ECG, and EEG, so as to overcome the limitations of existing railway driver psychological state monitoring methods and improve intelligent railway safety systems. To address the challenges of dynamically acquiring, accurately labeling, and interpreting SA states, the study proposed a systematic research method covering SA induction, annotation, modeling, and explanation, revealing the mechanism of multi-modal physiological features in SA recognition. To address the limited generalization capability of models caused by physiological signal non-stationarity, a domain generalization method with hybrid feature selection strategy was constructed, improving model adaptability under cross-subject and cross-session conditions. Additionally, to address the EEG signal acquisition limitations, a multi-modal learning method based on latent space modeling and conditional generation mechanism was developed, achieving robust SA recognition under EEG modality missing conditions. This research provides theoretical and methodological support for constructing highly reliable and adaptable train driving safety state monitoring systems, enhancing the intelligence and safety of train driving human-machine collaboration systems.
关键词
列车驾驶 /
情境意识 /
多模态生理信号 /
领域泛化 /
多模态学习
Key words
train driving /
situation awareness /
multimodal physiological signals /
domain generalization /
multimodal learning
{{custom_sec.title}}
{{custom_sec.title}}
{{custom_sec.content}}
参考文献
[1] WHITLOCK A, PETHICK J, MILLS A.Driver Vigilance Devices[M]//Rail Human Factors. London: Routledge, 2017: 120-130.
[2] KYRIAKIDIS M, MAJUMDAR A, OCHIENG W Y.Data Based Framework to Identify the Most Significant Performance Shaping Factors in Railway Operations[J]. Safety Science, 2015(78): 60-76.
[3] BAYSARI M T, CAPONECCHIA C, MCINTOSH A S, et al.Classification of Errors Contributing to Rail Incidents and Accidents: A Comparison of Two Human Error Identification Techniques[J]. Safety Science, 2009, 47(7): 948-957.
[4] ROTH E, MULTER J.Technology Implications of a Cognitive Task Analysis for Locomotive Engineers[R]. Washington: Federal Railroad Administration, 2009.
[5] ZHANG T, YANG J, LIANG N, et al.Physiological Measurements of Situation Awareness: A Systematic Review[J]. Human Factors, 2023, 65(5): 737-758.
[6] HASANZADEH S, ESMAEILI B, DODD M D.Examining the Relationship Between Construction Workers' Visual Attention and Situation Awareness Under Fall and Tripping Hazard Conditions: Using Mobile Eye Tracking[J]. Journal of Construction Engineering and Management, 2018, 144(7): 04018060.
[7] NGUYEN T, LIM C P, NGUYEN N D, et al.A Review of Situation Awareness Assessment Approaches in Aviation Environments[J]. IEEE Systems Journal, 2019, 13(3): 3590-3603.
[8] LIU W, XUE W.An Analysis of Situation Awareness for the Car Cab Display Interface Assessment Based on Driving Simulation[C]//Proceedings of the 4th International Conference on Electronics, Communications and Networks (CECNet 2014). Berlin: Springer, 2015: 310-315.
[9] WEI H, ZHUANG D, WANYAN X, et al.An Experimental Analysis of Situation Awareness for Cockpit Display Interface Evaluation Based on Flight Simulation[J]. Chinese Journal of Aeronautics, 2013, 26(4): 884-889.
[10] MEHTA R K, PERES S C, SHORTZ A E, et al.Operator Situation Awareness and Physiological States During Offshore Well Control Scenarios[J]. Journal of Loss Prevention in the Process Industries, 2018(55): 332-337.
[11] FENG C, LIU S, WANYAN X, et al.EEG Feature Analysis Related to Situation Awareness Assessment and Discrimination[J]. Aerospace, 2022, 9(10): 546.
[12] KANG Y, LIU F, CHEN W, et al.Recognizing Situation Awareness of Forklift Operators Based on Eye-Movement & EEG Features[J]. International Journal of Industrial Ergonomics, 2024(100): 103552.
[13] YEO L G, SUN H, LIU Y, et al.Mobile EEG-Based Situation Awareness Recognition for Air Traffic Controllers[C]//Proceedings of the 2017 IEEE International Conference on Systems, Man, and Cybernetics (SMC). Banff: IEEE, 2017: 721-726.
[14] SMEDEMARK-MARGULIES N.Reducing Calibration Effort for Brain-Computer Interfaces[D]. Boston: Northeastern University, 2024.
[15] GARGIULO G, CALVO R A, BIFULCO P, et al.A New EEG Recording System for Passive Dry Electrodes[J]. Clinical Neurophysiology, 2010, 121(5): 686-693.
[16] PATEL A N, JUNG T-P, SEJNOWSKI T J.A Wearable Multi-Modal Bio-Sensing System Towards Real-World Applications[J]. IEEE Transactions on Biomedical Engineering, 2018, 66(4): 1137-1147.
[17] IACOPI F, LIN C-T.A Perspective on Electroencephalography Sensors for Brain-Computer Interfaces[J]. Progress in Biomedical Engineering, 2022, 4(4): 043002.
[18] SIKANDER G, ANWAR S.Driver Fatigue Detection Systems: A Review[J]. IEEE Transactions on Intelligent Transportation Systems, 2018, 20(6): 2339-2352.
[19] BOBADE P, VANI M.Stress Detection with Machine Learning and Deep Learning Using Multimodal Physiological Data[C]//Proceedings of the 2020 Second International Conference on Inventive Research in Computing Applications (ICIRCA). Piscataway: IEEE, 2020: 450-455.
[20] MATTHEWS M D.Situation Awareness in a Virtual Environment: Description of a Subjective Assessment Scale[M]. Alexandria: US Army Research Institute for the Behavioral and Social Sciences, 2002.
[21] BANDT C, POMPE B.Permutation Entropy: A Natural Complexity Measure for Time Series[J]. Physical Review Letters, 2002, 88(17): 174102.
[22] KÄSTLE J L, ANVARI B, KROL J, et al. Correlation Between Situational Awareness and EEG Signals[J]. Neurocomputing, 2021(432): 70-79.
[23] TOPPI J, BORGHINI G, PETTI M, et al.Investigating Cooperative Behavior in Ecological Settings: An EEG Hyperscanning Study[J]. PloS One, 2016, 11(4): e0154236.
[24] BORGHINI G, ARICÒ P, DI FLUMERI G, et al.EEG-Based Cognitive Control Behaviour Assessment: An Ecological Study with Professional Air Traffic Controllers[J]. Scientific Reports, 2017, 7(1): 547.
[25] ELSSIED N O F, IBRAHIM O, OSMAN A H. A Novel Feature Selection Based on One-Way Anova F-Test for E-Mail Spam Classification[J]. Research Journal of Applied Sciences, Engineering and Technology, 2014, 7(3): 625-638.
[26] BLANCHARD G, LEE G, SCOTT C.Generalizing from Several Related Classification Tasks to a New Unlabeled Sample[J]. Advances in Neural Information Processing Systems, 2011(24): 39-45.
[27] ROY S, DORA S, MCCREADIE K, et al.MIEEG-GAN: Generating Artificial Motor Imagery Electroencephalography Signals[C]//Proceedings of the 2020 International Joint Conference on Neural Networks (IJCNN). Glasgow: IEEE, 2020: 1-8.
[28] ÖZDENIZCI O, WANG Y, KOIKE-AKINO T, et al.Learning Invariant Representations from EEG via Adversarial Inference[J]. IEEE Access, 2020(8): 27074-27085.
[29] YGER F, BERAR M, LOTTE F.Riemannian Approaches in Brain-Computer Interfaces: A Review[J]. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 2016, 25(10): 1753-1762.
[30] FDEZ J, GUTTENBERG N, WITKOWSKI O, et al.Cross-Subject EEG-Based Emotion Recognition Through Neural Networks with Stratified Normalization[J]. Frontiers in Neuroscience, 2021(15): 626277.
[31] CHEN L, ZHANG A, LOU X.Cross-Subject Driver Status Detection from Physiological Signals Based on Hybrid Feature Selection and Transfer Learning[J]. Expert Systems with Applications, 2019(137): 266-280.
[32] TU W, SUN S.Cross-Domain Representation-Learning Framework with Combination of Class-Separate and Domain-Merge Objectives[C]//Proceedings of the 1st International Workshop on Cross Domain Knowledge Discovery in Web and Social Network Mining. New York: ACM, 2012: 1-8.
[33] WU R, WANG H, CHEN H-T. A Comprehensive Survey on Deep Multimodal Learning with Missing Modality[J/OL]. arXiv, 2024, arXiv:2409.07825[2025-06-01]. Available: https://arxiv.org/abs/2409.07825
[34] REZA M K, PRATER-BENNETTE A, ASIF M S.Robust Multimodal Learning with Missing Modalities via Parameter-Efficient Adaptation[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024, 47(2): 742-754.
[35] WANG H, MA C, ZHANG J, et al.Learnable Cross-Modal Knowledge Distillation for Multi-Modal Learning with Missing Modality[C]//Proceedings of the International Conference on Medical Image Computing and Computer-Assisted Intervention (MICCAI). Cham: Springer, 2023: 456-467.
[36] WOO S, LEE S, PARK Y, et al.Towards Good Practices for Missing Modality Robust Action Recognition[C]//Proceedings of the AAAI Conference on Artificial Intelligence. Palo Alto: AAAI Press, 2023: 789-796.
[37] MA F, XU X, HUANG S-L, et al. Maximum Likelihood Estimation for Multimodal Learning with Missing Modality[J/OL]. arXiv, 2021, arXiv:2108.10513 (2025-03-13)[2025-03-14]. https://arxiv.org/abs/2108.10513.
基金
国家重点研发计划（2023YFF0615904）
PDF(2694 KB)
渝公网安备 50010702501717号