Assessment of Cognitive and Operational Effectiveness of Special Vehicle Crews Based on Multimodal Signals

LI Xinyan; CHENG Sijin; MA Siran; ZHANG Yijing; WEI Jianguo

doi:10.19554/j.cnki.1001-3563.2025.24.004

PDF(1869 KB)

Packaging Engineering ›› 2025, Vol. 46 ›› Issue (24) : 39-47. DOI: 10.19554/j.cnki.1001-3563.2025.24.004

Special Subject: Design of National Defense Equipment

Assessment of Cognitive and Operational Effectiveness of Special Vehicle Crews Based on Multimodal Signals

LI Xinyan^1,2, CHENG Sijin², MA Siran², ZHANG Yijing³, WEI Jianguo^1,*

Author information +

History +

Abstract

The work aims to systematically assess the cognitive and operational effectiveness of special vehicle crews under different interaction modes by leveraging multimodal signals. Experiments were conducted with a specialized vehicle simulation platform, incorporating three types of tasks: maneuvering, reconnaissance, and attack. Eight experienced participants performed these tasks under either an intelligent interaction mode or a conventional interaction mode. Data were collected from subjective reports (self-assessment scales of cognitive and operational capabilities), behavioral metrics (task completion time, interaction frequency) and electroencephalography (EEG) signals. The intelligent interaction group exhibited superior performance across all metrics, including subjective reports, behavioral data, and EEG signals, indicating significantly enhanced cognitive and operational capabilities compared to the conventional group. The findings suggest that multimodal signals can effectively reflect the cognitive and operational effectiveness of the crew, with the power value of the EEG signal in the theta frequency band offering certain advantages in assessing cognitive effectiveness.

Key words

special vehicle / multimodal signals / cognitive effectiveness / operational effectiveness

Cite this article

EndNote

Ris (Procite)

Bibtex

Download Citations

LI Xinyan, CHENG Sijin, MA Siran, ZHANG Yijing, WEI Jianguo. Assessment of Cognitive and Operational Effectiveness of Special Vehicle Crews Based on Multimodal Signals[J]. Packaging Engineering. 2025, 46(24): 39-47 https://doi.org/10.19554/j.cnki.1001-3563.2025.24.004

References

[1] 谢磊, 杜忠华, 王腾, 等. 独立转向与驱动无人装甲车操稳性分析[J]. 火力与指挥控制, 2019, 44(2): 21-25.
XIE L, DU Z H, WANG T, et al.Analysis on Steering Stability of Independent Steering and Driving of Unmanned Armored Vehicles[J]. Fire Control & Command Control, 2019, 44(2): 21-25.
[2] 陈成, 周云波, 张明. 装甲车用钢/纤维复合装甲防弹性能及影响因素研究[J]. 兵器装备工程学报, 2022, 43(5): 90-96.
CHEN C, ZHOU Y B, ZHANG M.Research on Anti-Ballistic Performance and Influencing Factors of Steel/Fiber Composite Armor for Armored Vehicles[J]. Journal of Ordnance Equipment Engineering, 2022, 43(5): 90-96.
[3] 方涛, 赵琴, 张官亮. 装甲战车隐身技术的研究现状及应用[J]. 兵器装备工程学报, 2023, 44(12): 133-140.
FANG T, ZHAO Q, ZHANG G L.Research Status and Application of Stealth Technology for Armored Fighting Vehicle[J]. Journal of Ordnance Equipment Engineering, 2023, 44(12): 133-140.
[4] 魏鹏, 王泽林, 尹渝, 等. 基于光纤传感的装甲车滚动轴承故障诊断技术[J]. 兵器装备工程学报, 2023, 44(1): 175-182.
WEI P, WANG Z L, YIN Y, et al.Fault Diagnosis Technology of the Rolling Bearing in an Armored Vehicle Based on Optical Fiber Sensors[J]. Journal of Ordnance Equipment Engineering, 2023, 44(1): 175-182.
[5] 郭司南, 完颜笑如, 刘双, 等. 智能化设计与信息加工通道复杂度对装甲车乘员脑力负荷的影响[J]. 兵工学报, 2021, 42(2): 234-241.
GUO S N, WANYAN X R, LIU S, et al.Influences of Intelligent Design and Information Processing Modality Complexity on Occupant Mental Workload[J]. Acta Armamentarii, 2021, 42(2): 234-241.
[6] 王建波, 沈卫, 胡阳旭. 2021年主要国家陆军装备技术发展综述[J]. 国防科技工业, 2022(1): 36-41.
WANG J B, SHEN W, HU Y X. Summary of the Development of Army Equipment Technology in Major Countries in2021[J]. Defence Science & Technology Industry, 2022(1): 36-41.
[7] CHAUVIN C, SAID F, LANGLOIS S.Augmented Reality HUD Vs. Conventional HUD to Perform a Navigation Task in a Complex Driving Situation[J]. Cognition, Technology & Work, 2023, 25(2): 217-232.
[8] CURRANO R, PARK S Y, MOORE D J, et al.Little Road Driving HUD: Heads-up Display Complexity Influences Drivers’ Perceptions of Automated Vehicles[C]//Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. Yokohama: ACM, 2021.
[9] 侯军芳, 王和平, 罗韬, 等. 装甲车辆舱内脉冲噪声评价方法[J]. 噪声与振动控制, 2016, 36(1): 79-82.
HOU J F, WANG H P, LUO T, et al.Evaluation Method for Internal Impulsive Noise in Armored Vehicles[J]. Noise and Vibration Control, 2016, 36(1): 79-82.
[10] 傅斌贺, 刘维平, 聂俊峰, 等. 考虑认知行为差异的乘员信息作业绩效研究[J]. 兵工学报, 2019, 40(3): 659-665.
FU B H, LIU W P, NIE J F, et al.Research on Crew's Information Operation Performance with the Difference of Cognitive Behavior[J]. Acta Armamentarii, 2019, 40(3): 659-665.
[11] MCNEESE N J, DEMIR M, COOKE N J, et al.Team Situation Awareness and Conflict: A Study of Human-Machine Teaming[J]. Journal of Cognitive Engineering and Decision Making, 2021, 15(2/3): 83-96.
[12] MICHAILOVS S, POND S, SCHMITT M, et al.The Impact of Information Integration in a Simulation of Future Submarine Command and Control[J]. Human Factors, 2023, 65(7): 1473-1490.
[13] VIRDI S S, NG Y T, LIU Y S, et al.Assessment of Situation Awareness for Seafarers Using Eye-Tracking Data[C]//Volume 1: Offshore Technology. Hamburg, Germany. American Society of Mechanical Engineers, 2022: 85857.
[14] YANG J, LIANG N, PITTS B J, et al.An Eye-Fixation Related Electroencephalography Technique for Predicting Situation Awareness: Implications for Driver State Monitoring Systems[J]. Hum Factors, 2024, 66(8): 2138-2153.
[15] DINGES D F, POWELL J W.Microcomputer Analyses of Performance on a Portable, Simple Visual RT Task during Sustained Operations[J]. Behavior Research Methods, Instruments, & Computers, 1985, 17(6): 652-655.
[16] RIANI K, PAPAKOSTAS M, KOKASH H, et al.Towards Detecting Levels of Alertness in Drivers Using Multiple Modalities[C]//Proceedings of the 13th ACM International Conference on PErvasive Technologies Related to Assistive Environments. Corfu: ACM, 2020: 1-9.
[17] HADRA M, OMIDVARNIA A, MESBAH M.Temporal Complexity of EEG Encodes Human Alertness[J]. Physiological Measurement, 2022, 43(9): 095002.
[18] AFFANNI A, NAJAFI T A.Drivers' Attention Assessment by Blink Rate Measurement from EEG Signals[C]//2022 IEEE International Workshop on Metrology for Automotive (MetroAutomotive). Modena: IEEE, 2022: 128-132.
[19] YAMASHITA J, TERASHIMA H, YONEYA M, et al.Pupillary Fluctuation Amplitude before Target Presentation Reflects Short-Term Vigilance Level in Psychomotor Vigilance Tasks[J]. PLoS One, 2021, 16(9): e0256953.
[20] HEMPEL A, GIESEL F L, GARCIA CARABALLO N M, et al. Plasticity of Cortical Activation Related to Working Memory during Training[J]. The American Journal of Psychiatry, 2004, 161(4): 745-747.
[21] SCHNEIDERS J A, OPITZ B, KRICK C M, et al.Separating Intra-Modal and Across-Modal Training Effects in Visual Working Memory: An fMRI Investigation[J]. Cerebral Cortex, 2011, 21(11): 2555-2564.
[22] PASSENBERG C, PEER A, BUSS M.A Survey of Environment-, Operator-, and Task-Adapted Controllers for Teleoperation Systems[J]. Mechatronics, 2010, 20(7): 787-801.
[23] AKAGI T M, SCHLEGEL R E, SHEHAB R L, et al.Toward the Construction of an Efficient Set of Robot Arm Operator Performance Metrics[J]. Proceedings of the Human Factors and Ergonomics Society Annual Meeting, 2004, 48(10): 1194-1198.
[24] KANGAS J, KUMAR S K, MEHTONEN H, et al.Trade-off between Task Accuracy, Task Completion Time and Naturalness for Direct Object Manipulation in Virtual Reality[J]. Multimodal Technologies and Interaction, 2022, 6(1): 6.
[25] SHAW L H, FREEDMAN E G, CROSSE M J, et al.Operating in a Multisensory Context: Assessing the Interplay between Multisensory Reaction Time Facilitation and Inter-Sensory Task-Switching Effects[J]. Neuroscience, 2020, 436: 122-135.
[26] BLOEMSAAT J G, MEULENBROEK R G J, VAN GALEN G P. Differential Effects of Mental Load on Proximal and Distal Arm Muscle Activity[J]. Experimental Brain Research, 2005, 167(4): 622-634.
[27] TREASTER D, MARRAS W S, BURR D, et al.Myofascial Trigger Point Development from Visual and Postural Stressors During Computer Work[J]. Journal of Electromyography and Kinesiology, 2006, 16(2): 115-124.
[28] PARSONS K C.Environmental Ergonomics: A Review of Principles, Methods and Models[J]. Applied Ergonomics, 2000, 31(6): 581-594.
[29] WIDIA M, DAWAL S Z M. The Effect of Hand-Held Vibrating Tools on Muscle Activity and Grip Strength[J]. Australian Journal of Basic and Applied Sciences, 2011, 5(11): 198-211.
[30] MAHDAVI N, DIANAT I, HEIDARIMOGHADAM R, et al.A Review of Work Environment Risk Factors Influencing Muscle Fatigue[J]. International Journal of Industrial Ergonomics, 2020, 80: 103028.
[31] WIXTED F, O’ SULLIVAN L.The Moderating Role of End-Tidal CO₂ on Upper Trapezius Muscle Activity in Response to Sustained Attention[J]. International Journal of Industrial Ergonomics, 2017, 61: 1-12.
[32] ALADDIN M F, JALIL N A A, GUAN N Y, et al. Evaluation of Human Discomfort from Combined Noise and Whole-Body Vibration in Passenger Vehicle[J]. International Journal of Automotive and Mechanical Engineering, 2019, 16(2): 6808-6824.
[33] ALTINSOY E, MARAVICH M M.Influence of Seat Vibration Frequency on Total Annoyance and Interaction Effects Caused by Simultaneous Noise and Seat Vibrations in Commercial Vehicles[J]. Vibration, 2022, 5(2): 183-199.
[34] HUANG Y, GRIFFIN M J.The Relative Discomfort of Noise and Vibration: Effects of Stimulus Duration[J]. Ergonomics, 2014, 57(8): 1244-1255.
[35] HUANG Y, JIANG W.The Effect of Exposure Duration on the Subjective Discomfort of Aircraft Cabin Noise[J]. Ergonomics, 2017, 60(1): 18-25.
[36] PILLAI P, BALASINGAM B, KIM Y H, et al.Eye-Gaze Metrics for Cognitive Load Detection on a Driving Simulator[J]. IEEE/ASME Transactions on Mechatronics, 2022, 27(4): 2134-2141.
[37] HOSSAIN D, SALIMULLAH S M, MAHMUDI R, et al.Cognitive Load Measurement Using Galvanic Skin Response for Listening Tasks[C]//2019 4th International Conference on Electrical Information and Communication Technology (EICT). Khulna: IEEE, 2019.
[38] DUCHOWSKI A T, KREJTZ K, KREJTZ I, et al.The Index of Pupillary Activity: Measuring Cognitive Loadvis- à-vis Task Difficulty with Pupil Oscillation[C]//Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. Montreal: ACM, 2018: 1-13.
[39] ORLANDI L, BROOKS B.Measuring Mental Workload and Physiological Reactions in Marine Pilots: Building Bridges towards Redlines of Performance[J]. Applied Ergonomics, 2018, 69: 74-92.
[40] ZHANG S Y, TIAN Y, WANG C H, et al.Target Selection by Gaze Pointing and Manual Confirmation: Performance Improved by Locking the Gaze Cursor[J]. Ergonomics, 2020, 63(7): 884-895.
[41] BLISSING B, BRUZELIUS F, ERIKSSON O.Driver Behavior in Mixed and Virtual Reality - a Comparative Study[J]. Transportation Research Part F: Traffic Psychology and Behaviour, 2019, 61: 229-237.
[42] DEBIE E, FERNANDEZ ROJAS R, FIDOCK J, et al.Multimodal Fusion for Objective Assessment of Cognitive Workload: A Review[J]. IEEE Transactions on Cybernetics, 2021, 51(3): 1542-1555.
[43] DING Y, CAO Y Q, DUFFY V G, et al.Measurement and Identification of Mental Workload during Simulated Computer Tasks with Multimodal Methods and Machine Learning[J]. Ergonomics, 2020, 63(7): 896-908.
[44] JIANG Z B, LI X Y, GE L Z, et al.Using Multimodal Methods and Machine Learning to Recognize Mental Workload: Distinguishing between Underload, Moderate Load, and Overload[J]. International Journal of Human-Computer Interaction, 2025, 41(8): 4742-4758.
[45] ALBERT B, TULLIS T.Measuring the User Experience[M]. 2nd ed. Beijing: Publishing House of Electronics Industry, 2024.