Situation Awareness Recognition Methods for Train Driving Based on Multimodal Physiological Data

DONG Wenli; FANG Weining; WANG Kun; NIU Ke

doi:10.19554/j.cnki.1001-3563.2025.16.003

PDF(2694 KB)

Packaging Engineering ›› 2025, Vol. 46 ›› Issue (16) : 33-44. DOI: 10.19554/j.cnki.1001-3563.2025.16.003

Special Subject: Innovation in Digital Intelligent Transportation Systems

Situation Awareness Recognition Methods for Train Driving Based on Multimodal Physiological Data

DONG Wenli^1a, FANG Weining^1b*, WANG Kun^1a, NIU Ke²

Author information +

History +

Abstract

Situation Awareness (SA), as a higher-order psychological variable reflecting train drivers' cognitive ability of the operating environment, provides a more comprehensive characterization of driving safety status compared with traditional single psychological state indicators. The work aims to conduct multi-modal recognition research on SA states based on three types of physiological data: eye movement, ECG, and EEG, so as to overcome the limitations of existing railway driver psychological state monitoring methods and improve intelligent railway safety systems. To address the challenges of dynamically acquiring, accurately labeling, and interpreting SA states, the study proposed a systematic research method covering SA induction, annotation, modeling, and explanation, revealing the mechanism of multi-modal physiological features in SA recognition. To address the limited generalization capability of models caused by physiological signal non-stationarity, a domain generalization method with hybrid feature selection strategy was constructed, improving model adaptability under cross-subject and cross-session conditions. Additionally, to address the EEG signal acquisition limitations, a multi-modal learning method based on latent space modeling and conditional generation mechanism was developed, achieving robust SA recognition under EEG modality missing conditions. This research provides theoretical and methodological support for constructing highly reliable and adaptable train driving safety state monitoring systems, enhancing the intelligence and safety of train driving human-machine collaboration systems.

Key words

train driving / situation awareness / multimodal physiological signals / domain generalization / multimodal learning

Cite this article

EndNote

Ris (Procite)

Bibtex

Download Citations

DONG Wenli, FANG Weining, WANG Kun, NIU Ke. Situation Awareness Recognition Methods for Train Driving Based on Multimodal Physiological Data[J]. Packaging Engineering. 2025, 46(16): 33-44 https://doi.org/10.19554/j.cnki.1001-3563.2025.16.003

References

[1] WHITLOCK A, PETHICK J, MILLS A.Driver Vigilance Devices[M]//Rail Human Factors. London: Routledge, 2017: 120-130.
[2] KYRIAKIDIS M, MAJUMDAR A, OCHIENG W Y.Data Based Framework to Identify the Most Significant Performance Shaping Factors in Railway Operations[J]. Safety Science, 2015(78): 60-76.
[3] BAYSARI M T, CAPONECCHIA C, MCINTOSH A S, et al.Classification of Errors Contributing to Rail Incidents and Accidents: A Comparison of Two Human Error Identification Techniques[J]. Safety Science, 2009, 47(7): 948-957.
[4] ROTH E, MULTER J.Technology Implications of a Cognitive Task Analysis for Locomotive Engineers[R]. Washington: Federal Railroad Administration, 2009.
[5] ZHANG T, YANG J, LIANG N, et al.Physiological Measurements of Situation Awareness: A Systematic Review[J]. Human Factors, 2023, 65(5): 737-758.
[6] HASANZADEH S, ESMAEILI B, DODD M D.Examining the Relationship Between Construction Workers' Visual Attention and Situation Awareness Under Fall and Tripping Hazard Conditions: Using Mobile Eye Tracking[J]. Journal of Construction Engineering and Management, 2018, 144(7): 04018060.
[7] NGUYEN T, LIM C P, NGUYEN N D, et al.A Review of Situation Awareness Assessment Approaches in Aviation Environments[J]. IEEE Systems Journal, 2019, 13(3): 3590-3603.
[8] LIU W, XUE W.An Analysis of Situation Awareness for the Car Cab Display Interface Assessment Based on Driving Simulation[C]//Proceedings of the 4th International Conference on Electronics, Communications and Networks (CECNet 2014). Berlin: Springer, 2015: 310-315.
[9] WEI H, ZHUANG D, WANYAN X, et al.An Experimental Analysis of Situation Awareness for Cockpit Display Interface Evaluation Based on Flight Simulation[J]. Chinese Journal of Aeronautics, 2013, 26(4): 884-889.
[10] MEHTA R K, PERES S C, SHORTZ A E, et al.Operator Situation Awareness and Physiological States During Offshore Well Control Scenarios[J]. Journal of Loss Prevention in the Process Industries, 2018(55): 332-337.
[11] FENG C, LIU S, WANYAN X, et al.EEG Feature Analysis Related to Situation Awareness Assessment and Discrimination[J]. Aerospace, 2022, 9(10): 546.
[12] KANG Y, LIU F, CHEN W, et al.Recognizing Situation Awareness of Forklift Operators Based on Eye-Movement & EEG Features[J]. International Journal of Industrial Ergonomics, 2024(100): 103552.
[13] YEO L G, SUN H, LIU Y, et al.Mobile EEG-Based Situation Awareness Recognition for Air Traffic Controllers[C]//Proceedings of the 2017 IEEE International Conference on Systems, Man, and Cybernetics (SMC). Banff: IEEE, 2017: 721-726.
[14] SMEDEMARK-MARGULIES N.Reducing Calibration Effort for Brain-Computer Interfaces[D]. Boston: Northeastern University, 2024.
[15] GARGIULO G, CALVO R A, BIFULCO P, et al.A New EEG Recording System for Passive Dry Electrodes[J]. Clinical Neurophysiology, 2010, 121(5): 686-693.
[16] PATEL A N, JUNG T-P, SEJNOWSKI T J.A Wearable Multi-Modal Bio-Sensing System Towards Real-World Applications[J]. IEEE Transactions on Biomedical Engineering, 2018, 66(4): 1137-1147.
[17] IACOPI F, LIN C-T.A Perspective on Electroencephalography Sensors for Brain-Computer Interfaces[J]. Progress in Biomedical Engineering, 2022, 4(4): 043002.
[18] SIKANDER G, ANWAR S.Driver Fatigue Detection Systems: A Review[J]. IEEE Transactions on Intelligent Transportation Systems, 2018, 20(6): 2339-2352.
[19] BOBADE P, VANI M.Stress Detection with Machine Learning and Deep Learning Using Multimodal Physiological Data[C]//Proceedings of the 2020 Second International Conference on Inventive Research in Computing Applications (ICIRCA). Piscataway: IEEE, 2020: 450-455.
[20] MATTHEWS M D.Situation Awareness in a Virtual Environment: Description of a Subjective Assessment Scale[M]. Alexandria: US Army Research Institute for the Behavioral and Social Sciences, 2002.
[21] BANDT C, POMPE B.Permutation Entropy: A Natural Complexity Measure for Time Series[J]. Physical Review Letters, 2002, 88(17): 174102.
[22] KÄSTLE J L, ANVARI B, KROL J, et al. Correlation Between Situational Awareness and EEG Signals[J]. Neurocomputing, 2021(432): 70-79.
[23] TOPPI J, BORGHINI G, PETTI M, et al.Investigating Cooperative Behavior in Ecological Settings: An EEG Hyperscanning Study[J]. PloS One, 2016, 11(4): e0154236.
[24] BORGHINI G, ARICÒ P, DI FLUMERI G, et al.EEG-Based Cognitive Control Behaviour Assessment: An Ecological Study with Professional Air Traffic Controllers[J]. Scientific Reports, 2017, 7(1): 547.
[25] ELSSIED N O F, IBRAHIM O, OSMAN A H. A Novel Feature Selection Based on One-Way Anova F-Test for E-Mail Spam Classification[J]. Research Journal of Applied Sciences, Engineering and Technology, 2014, 7(3): 625-638.
[26] BLANCHARD G, LEE G, SCOTT C.Generalizing from Several Related Classification Tasks to a New Unlabeled Sample[J]. Advances in Neural Information Processing Systems, 2011(24): 39-45.
[27] ROY S, DORA S, MCCREADIE K, et al.MIEEG-GAN: Generating Artificial Motor Imagery Electroencephalography Signals[C]//Proceedings of the 2020 International Joint Conference on Neural Networks (IJCNN). Glasgow: IEEE, 2020: 1-8.
[28] ÖZDENIZCI O, WANG Y, KOIKE-AKINO T, et al.Learning Invariant Representations from EEG via Adversarial Inference[J]. IEEE Access, 2020(8): 27074-27085.
[29] YGER F, BERAR M, LOTTE F.Riemannian Approaches in Brain-Computer Interfaces: A Review[J]. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 2016, 25(10): 1753-1762.
[30] FDEZ J, GUTTENBERG N, WITKOWSKI O, et al.Cross-Subject EEG-Based Emotion Recognition Through Neural Networks with Stratified Normalization[J]. Frontiers in Neuroscience, 2021(15): 626277.
[31] CHEN L, ZHANG A, LOU X.Cross-Subject Driver Status Detection from Physiological Signals Based on Hybrid Feature Selection and Transfer Learning[J]. Expert Systems with Applications, 2019(137): 266-280.
[32] TU W, SUN S.Cross-Domain Representation-Learning Framework with Combination of Class-Separate and Domain-Merge Objectives[C]//Proceedings of the 1st International Workshop on Cross Domain Knowledge Discovery in Web and Social Network Mining. New York: ACM, 2012: 1-8.
[33] WU R, WANG H, CHEN H-T. A Comprehensive Survey on Deep Multimodal Learning with Missing Modality[J/OL]. arXiv, 2024, arXiv:2409.07825[2025-06-01]. Available: https://arxiv.org/abs/2409.07825
[34] REZA M K, PRATER-BENNETTE A, ASIF M S.Robust Multimodal Learning with Missing Modalities via Parameter-Efficient Adaptation[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024, 47(2): 742-754.
[35] WANG H, MA C, ZHANG J, et al.Learnable Cross-Modal Knowledge Distillation for Multi-Modal Learning with Missing Modality[C]//Proceedings of the International Conference on Medical Image Computing and Computer-Assisted Intervention (MICCAI). Cham: Springer, 2023: 456-467.
[36] WOO S, LEE S, PARK Y, et al.Towards Good Practices for Missing Modality Robust Action Recognition[C]//Proceedings of the AAAI Conference on Artificial Intelligence. Palo Alto: AAAI Press, 2023: 789-796.
[37] MA F, XU X, HUANG S-L, et al. Maximum Likelihood Estimation for Multimodal Learning with Missing Modality[J/OL]. arXiv, 2021, arXiv:2108.10513 (2025-03-13)[2025-03-14]. https://arxiv.org/abs/2108.10513.