Gaze Interaction Design of Virtual Human Based on Joint Attention

ZHANG Ting; LAI Jiandou; ZHANG Jingjing; LI Jing; HOU Guanhua; PAN Yongsheng

doi:10.19554/j.cnki.1001-3563.2025.14.013

PDF(1245 KB)

Packaging Engineering ›› 2025, Vol. 46 ›› Issue (14) : 121-131. DOI: 10.19554/j.cnki.1001-3563.2025.14.013

Industrial Design

Gaze Interaction Design of Virtual Human Based on Joint Attention

ZHANG Ting^1,2, LAI Jiandou², ZHANG Jingjing², LI Jing¹, HOU Guanhua^3,*, PAN Yongsheng³

Author information +

History +

Abstract

The work aims to design a universal and dynamic model for virtual human gaze interaction, establish a gaze-mapping system between the user and the virtual human, and provide support for generating virtual digital humans capable of gaze interaction. Based on the theory of joint attention, a framework for mutual interaction between the virtual human and the user was established. Eye-tracking devices were utilized to measure the user's view field, thereby providing a data foundation for generating the virtual digital human's gaze. Further, a gaze interaction design model was developed across three dimensions of data collection, processing, and presentation of gaze, along with a proposed method for calculating the interactive movements of the virtual digital human's head (neck) and eyes. Ultimately, a virtual human teaching interaction scenario was constructed in Unity, experiments were designed, and the usability of the model was verified. Learner experience results demonstrated that virtual teaching agents with gaze interaction capabilities facilitated positive pseudosocial interactions and enhanced the learning experience. The design practice addressed the shortcomings in the virtual humans' perception and interaction with learners' gaze, thereby equipping them with the ability to engage in gaze interaction. The research findings offer a foundation for the design of human-intelligent gaze interaction applications in scenarios such as information transfer, emotional communication, and task collaboration, which can advance the design and production of virtual humans, and hold broad academic significance and promising application prospects.

Key words

virtual human / joint attention / gaze interaction / eye movement coupling / interaction model

Cite this article

EndNote

Ris (Procite)

Bibtex

Download Citations

ZHANG Ting, LAI Jiandou, ZHANG Jingjing, LI Jing, HOU Guanhua, PAN Yongsheng. Gaze Interaction Design of Virtual Human Based on Joint Attention[J]. Packaging Engineering. 2025, 46(14): 121-131 https://doi.org/10.19554/j.cnki.1001-3563.2025.14.013

References

[1] 侯文军, 卜瑶华, 刘聪林. 虚拟数字人: 元宇宙人际交互的技术性介质[J]. 传媒, 2023(4): 25-27.
HOU W J, BU Y H, LIU C L.Virtual Digital People: The Technical Medium of Interpersonal Interaction in the Meta-Universe[J]. Media, 2023(4): 25-27.
[2] SENJU A, JOHNSON M H.The Eye Contact Effect: Mechanisms and Development[J]. Trends in Cognitive Sciences, 2009, 13(3): 127-134.
[3] BOUCHER J D, PATTACINI U, LELONG A, et al.I Reach Faster when I See You Look: Gaze Effects in Human-Human and Human-Robot Face-to-Face Cooperation[J]. Frontiers in Neurorobotics, 2012, 6: 3.
[4] PEREIRA A, OERTEL C, FERMOSELLE L, et al.Responsive Joint Attention in Human-Robot Interaction[C]//2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Macau. IEEE, 2019: 1080-1087.
[5] 陈静, 张璐. 感受性视角下人智交互研究: 主题挖掘、方法识别、未来发展[J]. 情报理论与实践, 2024, 47(2): 98-106.
CHEN J, ZHANG L.Human-AI Interaction Research from a Receptivity Perspective: Topic Classification, Method Identification, and Future Development[J]. Information Studies (Theory & Application), 2024, 47(2): 98-106.
[6] 靖秋. 基于CiteSpace的国内虚拟数字人研究领域知识图谱分析[J]. 中阿科技论坛(中英文), 2023(8): 108-112.
JING Q.Study on Knowledge Graph Analysis of Domestic Virtual Digital Human Research Based on CiteSpace[J]. China-Arab States Science and Technology Forum, 2023(8): 108-112.
[7] 王晓慧, 覃京燕. 虚拟数字人手势交互设计[J]. 包装工程, 2021, 42(6): 46-52.
WANG X H, QIN J Y.Gesture Interaction Design of Virtual Human[J]. Packaging Engineering, 2021, 42(6): 46-52.
[8] 吴瑜, 戴鹳融. 机器人交互设计中参与式设计思维及方法研究[J]. 包装工程, 2023, 44(24): 91-100.
WU Y, DAI G R.Participatory Design Thinking and Method in HRI Design[J]. Packaging Engineering, 2023, 44(24): 91-100.
[9] 李光亚, 司占军. 基于Unity3D的数字虚拟人交互技术研究与应用[J]. 印刷与数字媒体技术研究, 2024(2): 123-134.
LI G Y, SI Z J.Research and Implementation of Digital Virtual Human Interaction Technology Based on Unity3D[J]. Printing and Digital Media Technology Study, 2024(2): 123-134.
[10] 严通. 基于ChatGPT语言模型的虚拟数字人语音交互应用[J]. 电视技术, 2023, 47(8): 182-186.
YAN T.Virtual Digital Person Voice Interaction Application Based on ChatGPT Language Model[J]. Video Engineering, 2023, 47(8): 182-186.
[11] BILLECI L, NARZISI A, TONACCI A, et al.An Integrated EEG and Eye-Tracking Approach for the Study of Responding and Initiating Joint Attention in Autism Spectrum Disorders[J]. Scientific Reports, 2017, 7(1): 13560.
[12] CHEVALIER P, KOMPATSIARI K, CIARDO F, et al.Examining Joint Attention with the Use of Humanoid Robots-a New Approach to Study Fundamental Mechanisms of Social Cognition[J]. Psychonomic Bulletin & Review, 2020, 27(2): 217-236.
[13] KLEINKE C L.Gaze and Eye Contact: A Research Review[J]. Psychological Bulletin, 1986, 100(1): 78-100.
[14] SEBANZ N, BEKKERING H, KNOBLICH G.Joint Action: Bodies and Minds Moving Together[J]. Trends in Cognitive Sciences, 2006, 10(2): 70-76.
[15] MUNDY P, NEWELL L. Attention, Joint Attention,Social Cognition[J]. Current Directions in Psychological Science, 2007, 16(5): 269-274.
[16] SCHMIDT R C, O’BRIEN B. Evaluating the Dynamics of Unintended Interpersonal Coordination[J]. Ecological Psychology, 1997, 9(3): 189-206.
[17] DURANTON C, GAUNET F.Behavioural Synchronization from an Ethological Perspective: Overview of Its Adaptive Value[J]. Adaptive Behavior, 2016, 24(3): 181-191.
[18] RICHARDSON D C, DALE R.Looking to Understand: The Coupling between Speakers' and Listeners' Eye Movements and Its Relationship to Discourse Comprehension[J]. Cognitive Science, 2005, 29(6): 1045-1060.
[19] RISTIC J, FRIESEN C K, KINGSTONE A.Are Eyes Special? It Depends on how You Look at It[J]. Psychonomic Bulletin & Review, 2002, 9(3): 507-513.
[20] CARUANA N, MCARTHUR G, WOOLGAR A, et al.Detecting Communicative Intent in a Computerised Test of Joint Attention[J]. PeerJ, 2017, 5: e2899.
[21] ZALAKE M, WOODWARD J, KAPOOR A, et al.Assessing the Impact of Virtual Human's Appearance on Users' Trust Levels[C]//Proceedings of the 18th International Conference on Intelligent Virtual Agents. Sydney: ACM, 2018: 329-330.
[22] GARAU M, SLATER M, BEE S, et al.The Impact of Eye Gaze on Communication Using Humanoid Avatars[C]//Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. Seattle: ACM, 2001: 309-316.
[23] WILMS M, SCHILBACH L, PFEIFFER U, et al.It's in Your Eyes—Using Gaze-Contingent Stimuli to Create Truly Interactive Paradigms for Social Cognitive and Affective Neuroscience[J]. Social Cognitive and Affective Neuroscience, 2010, 5(1): 98-107.
[24] FRISCHEN A, BAYLISS A P, TIPPER S P.Gaze Cueing of Attention: Visual Attention, Social Cognition, and Individual Differences[J]. Psychological Bulletin, 2007, 133(4): 694-724.
[25] KAMPIS D, SOUTHGATE V.Altercentric Cognition: How Others Influence Our Cognitive Processing[J]. Trends in Cognitive Sciences, 2020, 24(11): 945-959.
[26] BRUINSMA Y, KOEGEL R L, KOEGEL L K.Joint Attention and Children with Autism: A Review of the Literature[J]. Mental Retardation and Developmental Disabilities Research Reviews, 2004, 10(3): 169-175.
[27] 杰米·司迪恩. 国际经典设计教程: 交互设计[M]. 孔祥富, 王海洋, 译. 北京: 电子工业出版社, 2015: 9-38.
STEANE J.The Principles & Processes of Interactive Design[M]. KONG X F, WANG H Y, Translate. Beijing: Publishing House of Electronics Industry, 2015: 9-38.
[28] 马晓娜, 童倩倩, 李一含, 等. 基于多模态触觉与伪触觉反馈的智能设计触觉体验构建研究[J]. 装饰, 2022(9): 28-33.
MA X N, TONG Q Q, LI Y H, et al.Research on the Construction of Intelligent Design Haptic Experience Based on Multimodal Haptic and Pseudo-Haptic Feedback[J]. Zhuangshi, 2022(9): 28-33.
[29] RAYNER K, MILLER B, ROTELLO C M.Eye Movements when Looking at Print Advertisements: The Goal of the Viewer Matters[J]. Applied Cognitive Psychology, 2008, 22(5): 697-707.
[30] 黄莹, 王志良, 戚颖. 基于视线追踪的人机交互仿真系统的研制与应用[J]. 系统仿真学报, 2008, 20(24): 6678-6682.
HUANG Y, WANG Z L, QI Y.Development and Application of Human-Computer Interaction Simulation System Based on Gaze Tracking[J]. Journal of System Simulation, 2008, 20(24): 6678-6682.
[31] 詹泽慧. 基于智能Agent的远程学习者情感与认知识别模型——眼动追踪与表情识别技术支持下的耦合[J]. 现代远程教育研究, 2013, 25(5): 100-105.
ZHAN Z H.An Emotional and Cognitive Recognition Model for Distance Learners Based on Intelligent Agent—The Coupling of Eye Tracking and Expression Recognition Techniques[J]. Modern Distance Education Research, 2013, 25(5): 100-105.
[32] 蔡新元, 陶梦楚, 张健, 等. 信息呈现方式对学习效果的影响: 虚拟教师拟真度的调节作用[J]. 图书情报知识, 2023, 40(2): 20-28.
CAI X Y, TAO M C, ZHANG J, et al.The Impacts of Information Presentation Modes on Learning Effects: The Moderating Role of Virtual Teachers' Simulation Fidelity[J]. Documentation, Information & Knowledge, 2023, 40(2): 20-28.
[33] 陈凯泉. 实时导师和助手: 智能教学代理的实践定位[J]. 中国电化教育, 2006(4): 85-87.
CHEN K Q.Real-Time Tutor and Assistant: Practical Orientation of Intelligent Teaching Agent[J]. China Educational Technology, 2006(4): 85-87.
[34] BEEGE M, SCHNEIDER S, NEBEL S, et al.Look into My Eyes! Exploring the Effect of Addressing in Educational Videos[J]. Learning and Instruction, 2017, 49: 113-120.
[35] STULL A T, FIORELLA L, GAINER M J, et al.Using Transparent Whiteboards to Boost Learning from Online STEM Lectures[J]. Computers & Education, 2018, 120: 146-159.
[36] PI Z L, XU K, LIU C X, et al.Instructor Presence in Video Lectures: Eye Gaze Matters, but Not Body Orientation[J]. Computers & Education, 2020, 144: 103713.