Designing and Evaluating Gesture Interaction for AR-Based Industrial Inspections

ZHANG Fan; HUANG Xiaofeng; LOU Huanzhi; LI Tian

doi:10.19554/j.cnki.1001-3563.2026.04.004

PDF(1926 KB)

Packaging Engineering ›› 2026, Vol. 47 ›› Issue (4) : 39-52. DOI: 10.19554/j.cnki.1001-3563.2026.04.004

Industrial Design

Designing and Evaluating Gesture Interaction for AR-Based Industrial Inspections

ZHANG Fan, HUANG Xiaofeng, LOU Huanzhi, LI Tian^*

Author information +

History +

Abstract

The work aims to construct and verify a natural gesture interaction scheme that balances intuitive cognition with ergonomic comfort to address the cognitive disconnect in interaction and the physiological fatigue associated with prolonged operations in industrial pipe gallery inspections using Augmented Reality (AR) headsets. Firstly, integrating Reality-Based Interaction (RBI) and ergonomics theories, an "Intuition-Ergonomics" dual-constraint interaction design model was constructed. Then, through task analysis, the core requirements of pipe gallery inspection were defined, and a "Wizard of Oz" experiment was conducted to elicit users instinctive intents. To address the fatigue effects caused by large-amplitude arm movements, a "micro-gesture" correction strategy was introduced, establishing a gesture set containing seven core actions. Finally, a "context simulation" experimental platform featuring physical pipe props was developed to empirically evaluate the scheme's intuitiveness, comfort, and task robustness within a mixed reality environment. Experimental data indicated high user acceptance, with an average overall usability score of 4.35 (SD=0.65) and a "willingness to use" score reaching 4.49. Notably, the "fingertip micro-manipulation" strategy significantly enhanced perceived comfort, effectively reconciling the conflict between operational intuitiveness and physiological load. The context-adaptive micro-gesture interaction scheme proposed in this study successfully validates a complete design paradigm spanning theoretical construction, strategy revision, and contextualized evaluation, providing theoretical and empirical support for the practical application of AR technology in the field of industrial inspection.

Key words

augmented reality / gesture interaction / industrial inspection

Cite this article

EndNote

Ris (Procite)

Bibtex

Download Citations

ZHANG Fan, HUANG Xiaofeng, LOU Huanzhi, LI Tian. Designing and Evaluating Gesture Interaction for AR-Based Industrial Inspections[J]. Packaging Engineering. 2026, 47(4): 39-52 https://doi.org/10.19554/j.cnki.1001-3563.2026.04.004

References

[1] 王宇希, 张凤军, 刘越. 增强现实技术研究现状及发展趋势[J]. 科技导报, 2018, 36(10): 75-83.
WANG Y X, ZHANG F J, LIU Y.Augmented Reality Technology[J]. Science & Technology Review, 2018, 36(10): 75-83.
[2] 何明, 陈莹莹, 张斌, 等. AR技术在配电站房巡检业务中的应用研究[J]. 现代信息科技, 2019, 3(1): 175-176.
HE M, CHEN Y Y, ZHANG B, et al.Research on Application of AR Technology in Power Distribution Station Inspection[J]. Modern Information Technology, 2019, 3(1): 175-176.
[3] 李子阳, 张东胜, 杜邦. 基于AR的智能安全巡检系统[J]. 北京化工大学学报(自然科学版), 2022, 49(5): 59-66.
LI Z Y, ZHANG D S, DU Y B.An Intelligent Safety Inspection System Based on AR Technology[J]. Journal of Beijing University of Chemical Technology(Natural Science Edition), 2022, 49(5): 59-66.
[4] 李大勇, 杨畅, 张永伍, 等. 基于AR技术的变电站智能巡检系统设计与实现[J]. 微型电脑应用, 2020, 36(8): 92-94.
LI D Y, YANG C, ZHANG Y W, et al.Application of Remote Assistance System for Intelligent Inspection of Substation Based on the AR Technology[J]. Microcomputer Applications, 2020, 36(8): 92-94.
[5] 李仕, 卢俊阳, 张明威, 等. 大场景下自由手势交互技术[J]. 印刷与数字媒体技术研究, 2023(4): 244-251.
LI S, LU J Y, ZHANG M W, et al.Free Gesture Interaction Technology in Large Scenes[J]. Printing and Digital Media Technology Research, 2023(4): 244-251.
[6] 瞿珏, 焦浩洋, 王庆力, 等. 直感交互人机界面研究综述[J]. 兵器装备工程学报, 2023, 44(12): 141-150.
QU J, JIAO H Y, WANG Q L, et al.Review of Direct Interaction Human-Machine Interface Research[J]. Journal of Ordnance Equipment Engineering, 2023, 44(12): 141-150.
[7] 易心武, 薛锦云, 游珍, 等. 沉浸式虚拟现实多模态交互模型研究[J]. 江西师范大学学报(自然科学版), 2024, 48(1): 52-58.
YI X W, XUE J Y, YOU Z, et al.Research on Multimodal Interaction Model of Immersive Virtual Reality[J]. Journal of Jiangxi Normal University (Natural Science Edition), 2024, 48(1): 52-58.
[8] VERTEGAAL R.A Fitts Law Comparison of Eye Tracking and Manual Input in the Selection of Visual Targets[C]// Proceedings of the 10th International Conference on Multimodal Interfaces. New York, NY, USA: Association for Computing Machinery, 2008: 241-248.
[9] HURTIENNE J, KLÖCKNER K, DIEFENBACH S,et al. Designing with Image Schemas: Resolving the Tension between Innovation,Inclusion and Intuitive Use[J]. Interacting with Computers, 2015, 27(3): 235-255.
[10] YU C, SUN K, ZHONg M Y, et al.One-dimensional Handwriting: Inputting Letters and Words on Smart Glasses[C]// Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems. San Jose: CHI, 2016: 71-82
[11] 魏嘉焜, 王家润. 手势识别与交互综述[J]. 计算机与现代化, 2024(8): 67-76.
WEI J K, WANG J R.Survey on Gesture Recognition and Interaction[J]. Computer and Modernization, 2024(8): 67-76.
[12] QIAN X Y, HAN L,ZHANG Y L, et al.An Object Tracking Method Using Deep Learning and Adaptive Particle Filter for Night Fusion Image[C]// Proceedings of the 2017 International Conference on Progress in Informatics and Computing. Paris: IEEE, 2017: 138-142.
[13] 丛玉华, 何啸, 邢长达, 等. 基于计算机视觉手势识别的人机交互技术研究[J]. 兵器装备工程学报, 2022, 43(1): 152-160.
CONG Y H, HE X, XING C D, et al.Research on Human-Computer Interaction Technology Based on Computer Vision Gesture Recognition[J]. Journal of Ordnance Equipment Engineering, 2022, 43(1): 152-160.
[14] 王丽园, 赵陶, 王文, 等. 具有姿态变化鲁棒性的行人检测跟踪算法[J]. 计算机工程与设计, 2022, 43(10): 2877-2881.
WANG L Y, ZHAO T, WANG W, et al.Pedestrian Detection and Tracking Algorithm with Robustness to Pose Changes[J]. Computer Engineering and Design, 2022, 43(10): 2877-2881.
[15] 方新国, 王赞强, 高健权. 基于Leap Motion的手势交互系统设计[J]. 机械设计与研究, 2020, 36(2): 128-132.
FANG X G, WANG Z Q, GAO J Q.Design of Gesture Interaction System Based on Leap Motion[J]. Machine Design and Research, 2020, 36(2): 128-132.
[16] HINCAPIE-RAMOS J D, GUO X, MOGHADASIAN P, et al. Consumed Endurance: A Metric to Quantify Arm Fatigue of Mid-air Interactions[C]// CHI '14: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems.New York: Association for Computing Machinery, 2014: 1063-1072.
[17] WOBBROCK J O, MORRIS M R, WILSON A D.User-Defined Gestures for Surface Computing[C]// CHI '09: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. New York: Association for Computing Machinery, 2009: 1083-1092.
[18] PIUMSOMBOON T, CLARK A, BILLINGHURST M, et al.User-defined Gestures for Augmented Reality[C]// Proceedings of the CHI '13 Extended Abstracts on Human Factors in Computing Systems.New York: Association for Computing Machinery, 2013: 955-960.
[19] VILLARREAL-NARVAEZ S, VANDERDONCKT J, VATAVU R D, et al.A Systematic Review of Gesture Elicitation Studies: What can We Learn from 216 Studies?[C]// Proceedings of the 2020 ACM Designing Interactive systems Conference. New York: Association for Computing Machinery, 2020: 855-872.
[20] 钱晓松, 窦金花, 杜芃. 无人驾驶汽车行驶引导交互手势设计研究[J]. 包装工程, 2024, 45(8): 40-47.
QIAN X S, DOU J H, DU P.Research on Design of Interactive Gestures for Driving Guidance of Autonomous Vehicles[J]. Packaging Engineering, 2024, 45(8): 40-47.
[21] 鲁艺, 王晓烨, 王煜琳, 等. 面向混合现实教学的空中手势交互体验设计应用研究[J]. 包装工程, 2024, 45(4): 181-190.
LU Y, WANG X Y, WANG Y L, et al.Application of Mid-air Gesture Interactive Experience Design for Mixed Reality Teaching[J]. Packaging Engineering, 2024, 45(4): 181-190.
[22] 孙效华, 周博, 李彤. 隔空手势交互的设计要素与原则[J]. 包装工程, 2015, 36(8): 10-13.
SUN X H, ZHOU B, LI T.Design Elements and Principles of Mid-Air Gesture Interaction[J]. Packaging Engineering, 2015, 36(8): 10-13.
[23] 钱晓松, 张帆, 杨新. 基于全自动驾驶情境下驾驶导航需求的手势交互设计研究[J]. 装饰, 2020(4): 104-107.
QIAN X S, ZHANG F, YANG X.Study on Gesture Interaction for Navigation in The Autonomous Driving Situation[J]. Art & Design, 2020(4): 104-107.
[24] JACOB R J K, GIROUARD A,, HIRSHFIELD L M, et al. Reality-based Interaction: a Framework for Post-WIMP Interfaces[C]// Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. New York: Association for Computing Machinery, 2008: 201-210.
[25] NORMAN D A.The Design of Everyday Things[M]. New York: Basic Books, 2013.
[26] BOWMAN D A, KRUIJFF E, LAVIOLA J J J, et al. 3D User Interfaces: Theory and Practice[M]. Boston: Addison-Wesley, 2004.
[27] 陶建华, 巫英才, 喻纯, 等. 多模态人机交互综述[J]. 中国图象图形学报, 2022, 27(6): 1956-1987.
TAO J H, WU Y C, YU C, et al.A Survey on Multi-modal Human-computer Interaction[J]. Journal of Image and Graphics, 2022, 27(6): 1956-1987.
[28] MORRIS M R, WOBBROCK J O, WILSON A D.Understanding Users' Preferences for Surface Gestures[C]// Proceedings of Graphics Interface 2010. Toronto: Canadian Information Processing Society,2010: 261-268.