A Review on the Application Scenarios of "Vision +" Multi-sensory Experience Design under Synesthetic Vision

CHEN Hao; CHEN Wenwen; HU Guo; YOU Junhao

doi:10.19554/j.cnki.1001-3563.2025.24.027

PDF(451 KB)

Packaging Engineering ›› 2025, Vol. 46 ›› Issue (24) : 326-334. DOI: 10.19554/j.cnki.1001-3563.2025.24.027

Design Discussion

A Review on the Application Scenarios of "Vision +" Multi-sensory Experience Design under Synesthetic Vision

CHEN Hao¹, CHEN Wenwen^1,*, HU Guo¹, YOU Junhao²

Author information +

History +

Abstract

The multi-sensory experience design under the synesthetic vision is widely reflected as the "vision +" experience mode, which is mainly audio-visual synesthesia, and there are problems such as imbalance in the research of synesthetic interaction mode and narrow application scope. The work aims to systematically review the current research status of the application scenarios of synesthesia experience, so as to provide references for further exerting the application of synesthesia effect in experience design. Based on literature research and case analysis, the theoretical framework of synesthesia interaction mode was sorted out and summarized, and specific practical application fields were summarized by taking audio-visual, visual touch, visual taste, and visual olfactory synesthesia under the "vision +" mode as examples. In the multi-sensory experience design, the development of vision and hearing was relatively mature, and there were certain limitations in the research of the three low-level sensory channels of touch, taste and smell. Single sensory interaction is limited in immersive experience. The synesthetic experience is dominated by the "vision +" mode, and its sensory synergy is supported by the multi-sensory integration theory. Audio-visual synesthesia is the dominant mode of synesthetic interaction and is widely used in multimedia field. Visual touch, visual taste and visual smell show potential in advertising marketing and user experience, among which visual taste and visual smell synesthesia is more advantageous in the field of scene marketing and emotional arousal.

Key words

synesthesia / experience design / "vision +" / multi-sensory integration

Cite this article

EndNote

Ris (Procite)

Bibtex

Download Citations

CHEN Hao, CHEN Wenwen, HU Guo, YOU Junhao. A Review on the Application Scenarios of "Vision +" Multi-sensory Experience Design under Synesthetic Vision[J]. Packaging Engineering. 2025, 46(24): 326-334 https://doi.org/10.19554/j.cnki.1001-3563.2025.24.027

References

[1] 孙凌云, 王可幸, 陈培, 等. 多通道人机交互中感觉刺激的模拟方法研究综述[J]. 包装工程, 2023, 44(8): 81-94.
SUN L Y, WANG K X, CHEN P, et al.Research Review on Simulation Methods of Sensory Stimuli in Multi-channel Human-computer Interaction[J]. Packaging Engineering, 2023, 44(8): 81-94.
[2] 薛峰. 联觉体验视阈下的数字阅读界面设计方法探析[J]. 出版发行研究, 2019(9): 83-86.
XUE F.Analysis on the Design Method of Digital Reading Interface under the Visual Threshold of Synesthetic Experience[J]. Publication and Distribution Research, 2019(9): 83-86.
[3] JEWANSKI J, DAY S A, WARD J.A Colorful Albino: The First Documented Case of Synaesthesia, by Georg Tobias Ludwig Sachs in 1812[J]. Journal of the History of the Neurosciences, 2009, 18(3): 293-303.
[4] YONG Z, HSIEH P J, MILEA D.Seeing the Sound after Visual Loss: Functional MRI in Acquired Auditory-visual Synesthesia[J]. Experimental Brain Research, 2017, 235(2): 1-6.
[5] YANG M H, KRIEGMAN D J, AHUJA N.Detecting Faces in Images: A Survey[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2002, 24(1): 34-58.
[6] DENG L.Fundamentals of Speech Recognition[J]. LondonL: Prentice-Hall, Inc. 1993.
[7] 王韫, 李子晋. 无障碍设计研究:面向听障人群的振动触觉音乐体验[J]. 科技导报, 2023, 41(8): 74-82.
WANG Y, LI Z J.Research on Barrier-free design: Vibration-tactile Music Experience for Hearing-impaired People[J]. Science and Technology Review, 2023, 41(8): 74-82.
[8] 黄嘉丽, 黄宝华, 卢宇靖, 等. 电子舌检测技术及其在食品领域的应用研究进展[J]. 中国调味品, 2019, 44(5): 6.
HUANG J L, HUANG B H, LU Y J, et al.Research Progress of Electronic Tongue Detection Technology and Its Application in Food Field[J]. Chinese Condiments, 2019, 44(5): 6.
[9] 都甲洁, 宋钢. 将味道和气味数字化的传感器开发[J]. 中国酿造, 2017, 36(5): 1.
DU J J, SONG G.Development of Sensors to Digitize Taste and Odor[J]. China Brewing, 2017, 36(5): 1.
[10] 曾拓程, 王佳俊, 王晓玲, 等. 大场景视频监控下大坝运输车改进多目标多视觉卸料识别模型研究[J]. 水利学报, 2023, 54(5): 519-529.
ZENG T C, WANG J J, WANG X L, et al.Research on Improved Multi-objective Multi-vision Discharge identification Model of Dam transport Vehicle under Large Scene Video surveillance[J]. Journal of Hydraulic Engineering, 2023, 54(5): 519-529.
[11] 宋苏. 深度学习背景下计算机视觉算法应用探究——评《深度学习与计算机视觉: 算法原理, 框架应用与代码实现》[J]. 科技管理研究, 2024, 44(1): I0003.
SONG S.Application of Computer Vision Algorithms in the Context of Deep Learning-Review of Deep Learning and Computer Vision: Algorithm Principles, Framework Application and Code Implementation[J]. Science and Technology Management Research, 2024, 44(1): I0003.
[12] 刘岳开, 高宏力, 张洁. 基于开源项目的苹果采摘系统设计及视觉算法改进[J]. 江苏农业科学, 2019, 47(4): 6.
LIU Y K, GAO H L, ZHANG J.Apple picking system design and visual algorithm improvement based on open source project[J]. Jiangsu Agricultural Sciences, 2019, 47(4): 6.
[13] 张洁梅, 唐冰辛. 游戏化对虚拟CSR共创活动中消费者参与意愿的影响[J]. 中国软科学, 2022(9): 120-129.
ZHANG J M, TANG B X.The Impact of Gamification on Consumers' Participation Intention in Virtual CSR Co-creation[J]. China Soft Science, 2022(9): 120-129.
[14] 李星丽, 王践. 场景化与数字化: 广西民族博物馆陈列艺术探析[J]. 四川戏剧, 2020(4): 3.
LI X L, WANG J.Scenization and Digitalization: Analysis on Display Art in Guangxi Museum of Nationalities[J]. Sichuan Drama, 2020(4): 3.
[15] 梁永强, 李培林, 王崴, 等. 手机智能语音交互功能调查及对策[J]. 火力与指挥控制, 2019, 44(6): 6.
LIANG Y Q, LI P L, WANG W, et al.Investigation and Countermeasures of Intelligent Voice Interaction Function of Mobile Phone[J]. Fire Control & Command Control, 2019, 44(6): 6.
[16] 王晰巍, 刘宇桐, 乌吉斯古楞, 等. AI语音助手用户虚拟在线体验影响因素[J]. 图书馆论坛, 2024, 44(1): 71-85.
WANG X W, LIU Y T, WUGIS G L, et al.Factors Influencing Virtual Online Experience of AI Voice Assistant Users[J]. Library Forum, 2024, 44(1): 71-85.
[17] 杨文阳, 徐可欣, 闫辰祥. 基于VR的校园地震逃生沉浸式交互体验系统构建[J]. 中国电子科学研究院学报, 2023, 18(5): 407-413.
YANG W Y, XU K X, YAN C X.Construction of immersive interactive experience system for campus earthquake escape based on VR[J]. Journal of China Academy of Electronic Science, 2023, 18(5): 407-413.
[18] 王来顺, 何浩祥, 李少松. 基于智能手机的结构振动测试及舒适度改进评价方法[J]. 振动工程学报, 2023, 36(6): 1613-1622.
WANG L S, HE H X, LI S S.Structural Vibration Test and Comfort Improvement Evaluation Method based on Smartphone[J]. Journal of Vibration Engineering, 2023, 36(6): 1613-1622.
[19] 程龙, 刘泽宇. 柔性触觉传感技术及其在医疗康复机器人的应用[J]. 控制与决策, 2022, 37(6): 24.
CHENG L, LIU Z Y.Flexible tactile sensing technology and its application in Medical rehabilitation robot[J]. Control and Decision, 2022, 37(6): 24.
[20] 路奇, 吴昊, 梁婉, 等. 基于气味采集的嗅觉输入界面探索[J]. 计算机辅助设计与图形学学报, 2020, 32(7): 8.
LU Q, WU H, LIANG W, et al.Exploration of olfactory input interface based on odor acquisition[J]. Journal of Computer-Aided Design and Graphics, 2020, 32(7): 8.
[21] 姜艳. 后电影时代视听新媒体的"情动"生产, 呈现与传播[J]. 电影评介, 2023(14): 98-102.
JIANG Y.Production, Presentation and Dissemination of "Emotional Movement" in the post-film Era of Audiovisual New Media[J]. Film Review, 2023(14): 98-102.
[22] 周金超, 周尚晶. 联觉思维在鞋靴设计中的应用研究——以篮球运动鞋为例[J]. 皮革科学与工程, 2022, 32(3): 100-104.
ZHOU J C, ZHOU S J.Application of synesthetic thinking in Shoe design: A case study of basketball sports shoes[J]. Leather Science and Engineering, 2022, 32(3): 100-104.
[23] 曾栋, 周砖, 程海峰, 等. 产品设计中的触觉体验研究[J]. 包装工程, 2020, 41(2): 134-141.
ZENG D, ZHOU Z, CHENG H F, et al.Research on tactile experience in product design[J]. Packaging Engineering, 2020, 41(2): 134-141.
[24] 靳聪, 吴凤鹃, 李波, 等. 基于具身交互的听觉数字孪生及无人机作战应用[J]. 兵器装备工程学报, 2023, 44(12): 256-262.
JIN C, WU F J, LI B, et al.Auditory digital twin based on Embodied interaction and its application in UAV combat[J]. Journal of Ordnance Equipment Engineering, 2023, 44(12): 256-262.
[25] 胡晓斐, 汪嘉维, 刘涵宇, 等. 听觉刺激对虚拟环境中空间压缩的影响[J]. 心理学报, 2023, 55(1): 1-8.
HU X F, WANG J W, LIU H Y, et al.Effects of auditory stimuli on spatial compression in virtual environment[J]. Acta Psychologica Sinica, 2023, 55(1): 1-8.
[26] PALMER S E, SCHLOSS K B, XU Z, et al.Music-color associations are mediated by emotion[J]. Pnas, 2013, 110(22): 8836-8841.
[27] SPENCE C, DI STEFANO N.Coloured hearing, colour music, colour organs, and the search for perceptually meaningful correspondences between colour and sound[J]. I-Perception, 2022, 13(3): 20416695221092802.
[28] 刘姝彤. 电影音乐的视听联觉效应——浅析音乐在电影中的作用[J]. 电影新作, 2019(5): 119-122.
LIU S T.The Audiovisual synesthesia Effect of Film Music: Analysis of the role of music in film[J]. New Films, 2019(5): 119-122.
[29] 吴熙, 薛景文. 从二维到三维:绘本读物视听传播新路径[J]. 中国出版, 2024(7): 35-39.
WU X, XUE J W.From 2D to 3D: A New Audio-visual Communication Path for picture books[J]. China Publishing, 2024(7): 35-39.
[30] HAUCK P, VON CASTEN C, HECHT H.Crossmodal Correspondence between Music and Ambient Color Is Mediated by Emotion[J]. Multisensory Research, 2022, 35(5): 407-446.
[31] WANG W, LI Q, XIE J, et al.Research on emotional semantic retrieval of attention mechanism oriented to audio-visual synesthesia[J]. Neurocomputing, 2023(1): 20-30.
[32] HUANG J, ZHANG C, GUO L, et al.Exploring synesthesia-mapping mechanisms for music paintings based on machine learning[J]. Science, 2021(6563): 374.
[33] 郑浩. 游戏电影化叙事的视听美学与“影游融合”意识的重塑[J]. 电影评介, 2024(10): 36-40.
ZHENG H.The audiovisual aesthetics of cinematic narration in games and the reconstruction of consciousness of "Integration of film and game"[J]. Film Review, 2024(10): 36-40.
[34] 黄一伦, 朱嘉星, 代昊. 沉浸式视听技术在声景研究实践中的应用[J]. 复旦学报(自然科学版), 2024, 63(3): 329-335
HUANG Y L, ZHU J X, DAI H.Application of immersive audio-visual technology in soundscape research practice[J]. Journal of Fudan University (Natural Science Edition), 2024, 63(3): 329-335.
[35] 陆京生. 试论电视广告的视听结构[J]. 电视研究, 2000(4): 64-66.
LU J S.Discussion on Audiovisual Structure of TV Advertisement[J]. Television Research, 2000(4): 64-66.
[36] 冯胜勇. 网络视听文艺节目要“有魂、有根、有形、有容”[J]. 中国电视, 2024(8): 4-8.
FENG S Y.Discussion on Audiovisual Structure of TV Advertisement[J]. Television Research, 2000(4): 64-66.
[37] ZHANG J, ZHANG M, LIU Y, et al.Research on the Integration of Media Literacy Innovative Concept and Entrepreneurship Education and Digital Dynamic Creative Expression Talents[J]. Frontiers in psychology, 2021(19): 21-28.
[38] 李静雅, 任艺轩. 看见音乐:多模态交互技术在听障人群无障碍音乐体验中的设计应用[J].装饰, 2023(11): 142-144.
LI J Y, REN Y X.Seeing Music: Design and application of multimodal interaction technology in barrier-free music experience for hearing-impaired people[J]. Decoration, 2023(11): 142-144.
[39] 何聪艳, 胡吉永, 丁辛. 不同感觉模态下评价织物柔软感的心理物理特性[J]. 东华大学学报(自然科学版), 2012, 38(4): 381-385.
HE C Y, HU J Y, DING X.Psychophysical properties of fabric softness evaluation under different sensory modes[J]. Journal of Donghua University (Natural Science), 2012, 38(4): 381-385.
[40] 白小豆. 生成式人工智能语境中的视听新闻创新——基于ChatGPT技术应用的思考与展望[J]. 中国广播电视学刊, 2023(10): 5-8.
BAI X D.Audio-visual News Innovation in the Context of Generative Artificial Intelligence: Thinking and Prospect based on ChatGPT technology application[J]. Journal of China Radio and Television, 2023(10): 5-8.
[41] 祁欣. 高校视障学生应用型音乐人才的融合教育培养方法研究——评《中国特殊音乐教育历史与现状研究》[J]. 人民长江, 2022, 53(10): 248.
QI X.Research on Integrated Education Training Methods for Applied Music Talents of Visually impaired students in Colleges and Universities -- Review on the History and Current Situation of Special Music Education in China[J]. Yangtze River of China, 2019, 53(10): 248.
[42] NOEL C, DANDO R.The effect of emotional state on taste perception[J]. Appetite, 2015(95): 89-95.
[43] GUEDES D, PRADA M, LAMY E, et al.Bidirectionality in multisensory perception: Examining the mutual influences between audition and taste[J]. Food Quality and Preference, 2023(110): 104964.
[44] MOTOKI K, SPENCE C, VELASCO C.When visual cues influence taste/flavour perception: A systematic review[J]. Food Quality and Preference, 2023(111): 104996.
[45] SUGIMORI E, KAWASAKI Y.Cross-modal correspondence between visual information and taste perception of bitter foods and drinks[J]. Food Quality and Preference, 2022(98): 104539.
[46] CHEN Y, HUANG A X, FABER I, et al.Assessing the Influence of Visual-Taste Congruency on Perceived Sweetness and Product Liking in Immersive VR[J]. Foods, 2020, 9(4): 465.
[47] 梁金明. 基于视觉传达的食品包装创新设计研究--《食品包装设计》评述[J]. 食品与机械, 2021, 37(11): 241-242.
LIANG J M.Research on innovative design of Food Packaging based on visual communication -- Review of Food Packaging Design[J]. Food and Machinery, 201, 37(11): 241-242.
[48] 魏军. 音乐艺术与饮食文化的内在关系[J].食品研究与开发, 2019, 44(365): 88.
WEI J.The Intrinsic Relationship between Music Art and Food Culture[J]. Food Research and Development, 2019, 44(365): 88.
[49] 王帅. 感知个性化对顾客再购买意愿的作用机制和影响因素研究——基于在线教育平台的实证分析[D]. 北京: 对外经济贸易大学, 2021.
WANG S.Research on the mechanism and influencing factors of perceived personalization on customers' repurchase intention: An empirical analysis based on online education platform[D]. Beijing: University of International Business and Economics, 2021.
[50] 刘文彬, 漆正堂, 刘微娜.不同感觉功能对抑郁的影响及其神经机制[J].心理科学进展, 2023, 31(4): 641-656.
LIU W B, QI Z T, LIU W N.Effects of different sensory functions on depression and its neural mechanisms[J]. Advances in Psychological Science, 2019, 31(4): 641-656.
[51] HEIDARI F, SHIRAN M B, KAHENI H, et al.An fMRI-based investigation of the effects of odors on the functional connectivity network underlying the working memory[J]. Experimental Brain Research, 2024, 242(7): 31-40.
[52] 唐帮备, 朱明鑫, 邓凌云, 等. 基于嗅觉刺激的驾驶员负向情绪正向调控方法研究[J]. 汽车工程学报, 2023, 13(6): 832-843.
TANG B B, ZHU M X, DENG L Y, et al.Research on positive regulation of Negative Emotions in Drivers based on olfactory stimuli[J]. Journal of Automotive Engineering, 2023, 13(6): 832-843.
[53] 刘萍. 食品图像动态性对消费者食品感知和购买意愿的影响研究[D]. 成都: 四川大学, 2021.
LIU P.Study on the influence of food image dynamics on consumers' food perception and purchase intention [D]. Chengdu: Sichuan University, 2021.
[54] KOUBAA Y, ELEUCH A.Multimodal Perceptual Processing of Cues In Food Ads: Do You Smell What You See? Visual-Induced Olfactory Imagery and Its Effects On Taste Perception and Food Consumption[J]. Journal of Advertising Research, 2021, 61(1): 78-94.
[55] 孙辛欣, 周頔, 姜斌. 从嗅觉感知到形态表达的联觉设计实践:产品形态设计课程研究[J]. 装饰, 2022(4): 120-123.
SUN X X, ZHOU D, JIANG B.Synesthetic design Practice from olfactory perception to Morphological expression: A study on product morphological design curriculum[J]. Decoration, 2022(4): 120-123.
[56] BISWAS D, LABRECQUE L I, LEHMANN D R.Effects of Sequential Sensory Cues on Food Taste Perception: Cross㎝odal Interplay Between Visual and Olfactory Stimuli[J]. Journal of Consumer Psychology, 2021(19): 31-40.