基于多模态信号评估特种车辆乘员认知与操作效能

李欣颜; 程似锦; 马思然; 张艺竞; 魏建国

doi:10.19554/j.cnki.1001-3563.2025.24.004

PDF(1869 KB)

包装工程（设计栏目） ›› 2025, Vol. 46 ›› Issue (24) : 39-47. DOI: 10.19554/j.cnki.1001-3563.2025.24.004

专题:国防装备设计

基于多模态信号评估特种车辆乘员认知与操作效能

李欣颜^1,2, 程似锦², 马思然², 张艺竞³, 魏建国^1,*

作者信息 +

Assessment of Cognitive and Operational Effectiveness of Special Vehicle Crews Based on Multimodal Signals

LI Xinyan^1,2, CHENG Sijin², MA Siran², ZHANG Yijing³, WEI Jianguo^1,*

Author information +

文章历史 +

摘要

目的通过整合多模态信号,系统评估特种车辆乘员在不同交互模式下的认知与操作效能。方法基于某特种车辆仿真验证平台开展实验,8名有相关经验的被试依次以智能交互模式和普通交互2种模式执行侦察、机动和打击三类任务,同步采集了被试的主观报告（认知与操作能力自评量表）、行为（任务完成时间、任务交互次数）和脑电数据。结果智能交互组的主观报告、行为和脑电数据均优于普通交互组,其认知与操作能力显著提高。结论多模态信号能较好反映乘员认知与操作效能,其中脑电信号theta频段功率值在反映乘员认知效能中具有一定优势。

Abstract

The work aims to systematically assess the cognitive and operational effectiveness of special vehicle crews under different interaction modes by leveraging multimodal signals. Experiments were conducted with a specialized vehicle simulation platform, incorporating three types of tasks: maneuvering, reconnaissance, and attack. Eight experienced participants performed these tasks under either an intelligent interaction mode or a conventional interaction mode. Data were collected from subjective reports (self-assessment scales of cognitive and operational capabilities), behavioral metrics (task completion time, interaction frequency) and electroencephalography (EEG) signals. The intelligent interaction group exhibited superior performance across all metrics, including subjective reports, behavioral data, and EEG signals, indicating significantly enhanced cognitive and operational capabilities compared to the conventional group. The findings suggest that multimodal signals can effectively reflect the cognitive and operational effectiveness of the crew, with the power value of the EEG signal in the theta frequency band offering certain advantages in assessing cognitive effectiveness.

导出引用1

李欣颜, 程似锦, 马思然, 张艺竞, 魏建国. 基于多模态信号评估特种车辆乘员认知与操作效能[J]. 包装工程. 2025, 46(24): 39-47 https://doi.org/10.19554/j.cnki.1001-3563.2025.24.004

LI Xinyan, CHENG Sijin, MA Siran, ZHANG Yijing, WEI Jianguo. Assessment of Cognitive and Operational Effectiveness of Special Vehicle Crews Based on Multimodal Signals[J]. Packaging Engineering. 2025, 46(24): 39-47 https://doi.org/10.19554/j.cnki.1001-3563.2025.24.004

中图分类号： TB472

参考文献

[1] 谢磊, 杜忠华, 王腾, 等. 独立转向与驱动无人装甲车操稳性分析[J]. 火力与指挥控制, 2019, 44(2): 21-25.
XIE L, DU Z H, WANG T, et al.Analysis on Steering Stability of Independent Steering and Driving of Unmanned Armored Vehicles[J]. Fire Control & Command Control, 2019, 44(2): 21-25.
[2] 陈成, 周云波, 张明. 装甲车用钢/纤维复合装甲防弹性能及影响因素研究[J]. 兵器装备工程学报, 2022, 43(5): 90-96.
CHEN C, ZHOU Y B, ZHANG M.Research on Anti-Ballistic Performance and Influencing Factors of Steel/Fiber Composite Armor for Armored Vehicles[J]. Journal of Ordnance Equipment Engineering, 2022, 43(5): 90-96.
[3] 方涛, 赵琴, 张官亮. 装甲战车隐身技术的研究现状及应用[J]. 兵器装备工程学报, 2023, 44(12): 133-140.
FANG T, ZHAO Q, ZHANG G L.Research Status and Application of Stealth Technology for Armored Fighting Vehicle[J]. Journal of Ordnance Equipment Engineering, 2023, 44(12): 133-140.
[4] 魏鹏, 王泽林, 尹渝, 等. 基于光纤传感的装甲车滚动轴承故障诊断技术[J]. 兵器装备工程学报, 2023, 44(1): 175-182.
WEI P, WANG Z L, YIN Y, et al.Fault Diagnosis Technology of the Rolling Bearing in an Armored Vehicle Based on Optical Fiber Sensors[J]. Journal of Ordnance Equipment Engineering, 2023, 44(1): 175-182.
[5] 郭司南, 完颜笑如, 刘双, 等. 智能化设计与信息加工通道复杂度对装甲车乘员脑力负荷的影响[J]. 兵工学报, 2021, 42(2): 234-241.
GUO S N, WANYAN X R, LIU S, et al.Influences of Intelligent Design and Information Processing Modality Complexity on Occupant Mental Workload[J]. Acta Armamentarii, 2021, 42(2): 234-241.
[6] 王建波, 沈卫, 胡阳旭. 2021年主要国家陆军装备技术发展综述[J]. 国防科技工业, 2022(1): 36-41.
WANG J B, SHEN W, HU Y X. Summary of the Development of Army Equipment Technology in Major Countries in2021[J]. Defence Science & Technology Industry, 2022(1): 36-41.
[7] CHAUVIN C, SAID F, LANGLOIS S.Augmented Reality HUD Vs. Conventional HUD to Perform a Navigation Task in a Complex Driving Situation[J]. Cognition, Technology & Work, 2023, 25(2): 217-232.
[8] CURRANO R, PARK S Y, MOORE D J, et al.Little Road Driving HUD: Heads-up Display Complexity Influences Drivers’ Perceptions of Automated Vehicles[C]//Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. Yokohama: ACM, 2021.
[9] 侯军芳, 王和平, 罗韬, 等. 装甲车辆舱内脉冲噪声评价方法[J]. 噪声与振动控制, 2016, 36(1): 79-82.
HOU J F, WANG H P, LUO T, et al.Evaluation Method for Internal Impulsive Noise in Armored Vehicles[J]. Noise and Vibration Control, 2016, 36(1): 79-82.
[10] 傅斌贺, 刘维平, 聂俊峰, 等. 考虑认知行为差异的乘员信息作业绩效研究[J]. 兵工学报, 2019, 40(3): 659-665.
FU B H, LIU W P, NIE J F, et al.Research on Crew's Information Operation Performance with the Difference of Cognitive Behavior[J]. Acta Armamentarii, 2019, 40(3): 659-665.
[11] MCNEESE N J, DEMIR M, COOKE N J, et al.Team Situation Awareness and Conflict: A Study of Human-Machine Teaming[J]. Journal of Cognitive Engineering and Decision Making, 2021, 15(2/3): 83-96.
[12] MICHAILOVS S, POND S, SCHMITT M, et al.The Impact of Information Integration in a Simulation of Future Submarine Command and Control[J]. Human Factors, 2023, 65(7): 1473-1490.
[13] VIRDI S S, NG Y T, LIU Y S, et al.Assessment of Situation Awareness for Seafarers Using Eye-Tracking Data[C]//Volume 1: Offshore Technology. Hamburg, Germany. American Society of Mechanical Engineers, 2022: 85857.
[14] YANG J, LIANG N, PITTS B J, et al.An Eye-Fixation Related Electroencephalography Technique for Predicting Situation Awareness: Implications for Driver State Monitoring Systems[J]. Hum Factors, 2024, 66(8): 2138-2153.
[15] DINGES D F, POWELL J W.Microcomputer Analyses of Performance on a Portable, Simple Visual RT Task during Sustained Operations[J]. Behavior Research Methods, Instruments, & Computers, 1985, 17(6): 652-655.
[16] RIANI K, PAPAKOSTAS M, KOKASH H, et al.Towards Detecting Levels of Alertness in Drivers Using Multiple Modalities[C]//Proceedings of the 13th ACM International Conference on PErvasive Technologies Related to Assistive Environments. Corfu: ACM, 2020: 1-9.
[17] HADRA M, OMIDVARNIA A, MESBAH M.Temporal Complexity of EEG Encodes Human Alertness[J]. Physiological Measurement, 2022, 43(9): 095002.
[18] AFFANNI A, NAJAFI T A.Drivers' Attention Assessment by Blink Rate Measurement from EEG Signals[C]//2022 IEEE International Workshop on Metrology for Automotive (MetroAutomotive). Modena: IEEE, 2022: 128-132.
[19] YAMASHITA J, TERASHIMA H, YONEYA M, et al.Pupillary Fluctuation Amplitude before Target Presentation Reflects Short-Term Vigilance Level in Psychomotor Vigilance Tasks[J]. PLoS One, 2021, 16(9): e0256953.
[20] HEMPEL A, GIESEL F L, GARCIA CARABALLO N M, et al. Plasticity of Cortical Activation Related to Working Memory during Training[J]. The American Journal of Psychiatry, 2004, 161(4): 745-747.
[21] SCHNEIDERS J A, OPITZ B, KRICK C M, et al.Separating Intra-Modal and Across-Modal Training Effects in Visual Working Memory: An fMRI Investigation[J]. Cerebral Cortex, 2011, 21(11): 2555-2564.
[22] PASSENBERG C, PEER A, BUSS M.A Survey of Environment-, Operator-, and Task-Adapted Controllers for Teleoperation Systems[J]. Mechatronics, 2010, 20(7): 787-801.
[23] AKAGI T M, SCHLEGEL R E, SHEHAB R L, et al.Toward the Construction of an Efficient Set of Robot Arm Operator Performance Metrics[J]. Proceedings of the Human Factors and Ergonomics Society Annual Meeting, 2004, 48(10): 1194-1198.
[24] KANGAS J, KUMAR S K, MEHTONEN H, et al.Trade-off between Task Accuracy, Task Completion Time and Naturalness for Direct Object Manipulation in Virtual Reality[J]. Multimodal Technologies and Interaction, 2022, 6(1): 6.
[25] SHAW L H, FREEDMAN E G, CROSSE M J, et al.Operating in a Multisensory Context: Assessing the Interplay between Multisensory Reaction Time Facilitation and Inter-Sensory Task-Switching Effects[J]. Neuroscience, 2020, 436: 122-135.
[26] BLOEMSAAT J G, MEULENBROEK R G J, VAN GALEN G P. Differential Effects of Mental Load on Proximal and Distal Arm Muscle Activity[J]. Experimental Brain Research, 2005, 167(4): 622-634.
[27] TREASTER D, MARRAS W S, BURR D, et al.Myofascial Trigger Point Development from Visual and Postural Stressors During Computer Work[J]. Journal of Electromyography and Kinesiology, 2006, 16(2): 115-124.
[28] PARSONS K C.Environmental Ergonomics: A Review of Principles, Methods and Models[J]. Applied Ergonomics, 2000, 31(6): 581-594.
[29] WIDIA M, DAWAL S Z M. The Effect of Hand-Held Vibrating Tools on Muscle Activity and Grip Strength[J]. Australian Journal of Basic and Applied Sciences, 2011, 5(11): 198-211.
[30] MAHDAVI N, DIANAT I, HEIDARIMOGHADAM R, et al.A Review of Work Environment Risk Factors Influencing Muscle Fatigue[J]. International Journal of Industrial Ergonomics, 2020, 80: 103028.
[31] WIXTED F, O’ SULLIVAN L.The Moderating Role of End-Tidal CO₂ on Upper Trapezius Muscle Activity in Response to Sustained Attention[J]. International Journal of Industrial Ergonomics, 2017, 61: 1-12.
[32] ALADDIN M F, JALIL N A A, GUAN N Y, et al. Evaluation of Human Discomfort from Combined Noise and Whole-Body Vibration in Passenger Vehicle[J]. International Journal of Automotive and Mechanical Engineering, 2019, 16(2): 6808-6824.
[33] ALTINSOY E, MARAVICH M M.Influence of Seat Vibration Frequency on Total Annoyance and Interaction Effects Caused by Simultaneous Noise and Seat Vibrations in Commercial Vehicles[J]. Vibration, 2022, 5(2): 183-199.
[34] HUANG Y, GRIFFIN M J.The Relative Discomfort of Noise and Vibration: Effects of Stimulus Duration[J]. Ergonomics, 2014, 57(8): 1244-1255.
[35] HUANG Y, JIANG W.The Effect of Exposure Duration on the Subjective Discomfort of Aircraft Cabin Noise[J]. Ergonomics, 2017, 60(1): 18-25.
[36] PILLAI P, BALASINGAM B, KIM Y H, et al.Eye-Gaze Metrics for Cognitive Load Detection on a Driving Simulator[J]. IEEE/ASME Transactions on Mechatronics, 2022, 27(4): 2134-2141.
[37] HOSSAIN D, SALIMULLAH S M, MAHMUDI R, et al.Cognitive Load Measurement Using Galvanic Skin Response for Listening Tasks[C]//2019 4th International Conference on Electrical Information and Communication Technology (EICT). Khulna: IEEE, 2019.
[38] DUCHOWSKI A T, KREJTZ K, KREJTZ I, et al.The Index of Pupillary Activity: Measuring Cognitive Loadvis- à-vis Task Difficulty with Pupil Oscillation[C]//Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. Montreal: ACM, 2018: 1-13.
[39] ORLANDI L, BROOKS B.Measuring Mental Workload and Physiological Reactions in Marine Pilots: Building Bridges towards Redlines of Performance[J]. Applied Ergonomics, 2018, 69: 74-92.
[40] ZHANG S Y, TIAN Y, WANG C H, et al.Target Selection by Gaze Pointing and Manual Confirmation: Performance Improved by Locking the Gaze Cursor[J]. Ergonomics, 2020, 63(7): 884-895.
[41] BLISSING B, BRUZELIUS F, ERIKSSON O.Driver Behavior in Mixed and Virtual Reality - a Comparative Study[J]. Transportation Research Part F: Traffic Psychology and Behaviour, 2019, 61: 229-237.
[42] DEBIE E, FERNANDEZ ROJAS R, FIDOCK J, et al.Multimodal Fusion for Objective Assessment of Cognitive Workload: A Review[J]. IEEE Transactions on Cybernetics, 2021, 51(3): 1542-1555.
[43] DING Y, CAO Y Q, DUFFY V G, et al.Measurement and Identification of Mental Workload during Simulated Computer Tasks with Multimodal Methods and Machine Learning[J]. Ergonomics, 2020, 63(7): 896-908.
[44] JIANG Z B, LI X Y, GE L Z, et al.Using Multimodal Methods and Machine Learning to Recognize Mental Workload: Distinguishing between Underload, Moderate Load, and Overload[J]. International Journal of Human-Computer Interaction, 2025, 41(8): 4742-4758.
[45] ALBERT B, TULLIS T.Measuring the User Experience[M]. 2nd ed. Beijing: Publishing House of Electronics Industry, 2024.