Multimodal Interaction Design of Qiu Ying's Shanglin Garden Based on the Theory of Mediated Translation

ZHANG Xiaoxi; ZUO Peiqi

doi:10.19554/j.cnki.1001-3563.2025.18.041

PDF(28527 KB)

Packaging Engineering ›› 2025, Vol. 46 ›› Issue (18) : 460-469. DOI: 10.19554/j.cnki.1001-3563.2025.18.041

Design Discussion

Multimodal Interaction Design of Qiu Ying's Shanglin Garden Based on the Theory of Mediated Translation

ZHANG Xiaoxi, ZUO Peiqi

Author information +

History +

Abstract

The work aims to use digital technology and multimodal interaction as a means to deconstruct the "Shanglin Scroll" by Qiu Ying, a famous Ming Dynasty painter, and explore the path of cultural heritage protection and innovative inheritance. Based on the theory of "mediated translation", by analyzing and summarizing the information in Qiu Ying's "Shanglin Scroll" and Sima Xiangru's "Shanglin Prose", the multimodal interaction technology was combined to transform the advantages of cultural resources into contemporary development momentum. Through graphic analysis, mediated translation and multimodal interaction, the digital guide design system, multimodal immersive design and cultural and creative products with Shanglin Garden culture as the core were constructed. The digital and physical product models of the "one pond and three mountains" scene of Jianzhang Palace in the scroll were reproduced, and a guide system covering seven narrative nodes and a cross-modal collaborative interaction framework were created, which condensed thousands of years of ecological wisdom in the palm of hand and realized the visual presentation and interactive perception of excellent cultural heritage. A design method of "graphic and text deconstruction - media translation - modal interaction - scenario application" has been formed, and a research path driven by multimodal technology has been established, which provides theoretical support and practical reference for the innovative inheritance of cultural heritage in the digital age, so as to promote the innovative transformation of excellent cultural heritage into modern interactive experience.

Key words

Qiu Ying's Shanglin Garden / multimodal interaction design / mediated translation / heritage digitization / cultural and creative products

Cite this article

EndNote

Ris (Procite)

Bibtex

Download Citations

ZHANG Xiaoxi, ZUO Peiqi. Multimodal Interaction Design of Qiu Ying's Shanglin Garden Based on the Theory of Mediated Translation[J]. Packaging Engineering. 2025, 46(18): 460-469 https://doi.org/10.19554/j.cnki.1001-3563.2025.18.041

References

[1] 张云中, 姚佳颖, 冯晓. 文化遗产数字资源的数据生产与再生产理论探析[J/OL]. 图书馆杂志, 1-13[2025-04-22] (2025-09-11). https://link.cnki.net/urlid/31.1108.G2.20250422.0826.002.
ZHANG Y Z, YAO J Y, FENG X. Analysis on the Theory of Data Production and Reproduction of Cultural Heritage Digital Resources[J/OL]. Library Journal, 1-13[2025-04-22] (2025-09-11). https://link.cnki.net/urlid/31.1108.G2.20250422.0826.002.
[2] 肖建莉, 白智成, 穆星宇. 遗产数字化: 概念议题、研究启示与应用实践[J]. 城市规划学刊, 2024(6): 104-112.
XIAO J L, BAI Z C, MU X Y.Heritage Digitization: Conceptual Issues, Research Insights, and Application Practices[J]. Urban Planning Forum, 2024(6): 104-112.
[3] 台北故宫博物院编辑委员会. 故宫书画图录[M]. 台北: 台北故宫博物院, 1999.
Editorial Committee of the Taipei Palace Museum. A Catalogue of Calligraphy and Paintings in the Forbidden City[M]. Taipei: Taipei Palace Museum, 1999.
[4] 萧统, 李善. 文选[M]. 北京: 中华书局, 1977.
XIAO T, LI S.Selected Works[M]. Beijing: Zhonghua Book Company, 1977.
[5] 马晓娜, 张雨欣, 于茜. 基于多模态信息交互的智能家居设计研究[J]. 包装工程, 2022, 43(16): 59-115.
MA X N, ZHANG Y X, YU Q.Smart Home Design Based on Multimodal Information Interaction[J]. Packaging Engineering, 2022, 43(16): 59-115.
[6] BOLT R A."Put-that-there": Voice and Gesture at the Graphics Interface[C]//ACM Siggraph, New York:1980, 262-270.
[7] MUHAMMAD NIZAM S S, ZAINAL ABIDIN R, CHE HASHIM N, et al. A Review of Multimodal Interaction Technique in Augmented Reality Environment[J]. International Journal on Advanced Science, Engineering and Information Technology, 2018, 8(4-2): 1460-1469.
[8] LAZARO M J, LEE J, CHUN J, et al.Multimodal Interaction: Input-Output Modality Combinations for Identification Tasks in Augmented Reality[J]. Applied Ergonomics, 2022, 105: 103842.
[9] RAKKOLAINEN I, FAROOQ A, KANGAS J, et al.Technologies for Multimodal Interaction in Extended Reality—A Scoping Review[J]. Multimodal Technologies and Interaction, 2021, 5(12): 81.
[10] 陶建华, 巫英才, 喻纯, 等. 多模态人机交互综述[J]. 中国图象图形学报, 2022, 27(6): 1956-1987.
TAO J H, WU Y C, YU C, et al.A Survey on Multi- Modal Human-Computer Interaction[J]. Journal of Image and Graphics, 2022, 27(6): 1956-1987.
[11] 赵星植. 符号作为媒介: 论皮尔斯后期的传播哲学思想[J]. 符号与传媒, 2025(1): 65-78.
ZHAO X Z.Sign as the Medium: On Peirce's Later Philosophy of Communication[J]. Signs and Media, 2025(1): 65-78.
[12] 刘涛. 超越“象征之殇”: 物质逻辑与图像阐释的媒介视角——通往以媒介为方法的图像阐释学[J]. 探索与争鸣, 2022(12): 57-215.
LIU T.Beyond "Symbol Analysis": Media Materiality and Iconology from the Perspective of Media Theories[J]. Exploration and Free Views, 2022(12): 57-215.
[13] 沙垚, 杨风云. 乡村治理的中介化逻辑: 基于江苏省M村农文旅运营项目的田野调查[J]. 国际新闻界, 2025, 47(3): 26-45.
SHA Y, YANG F Y.The Mediated Logic of Rural Governance: Based on a Field Study of an Agri-Tourism Project in M Village, Jiangsu Province[J]. Chinese Journal of Journalism & Communication, 2025, 47(3): 26-45.
[14] 徐迎庆, 图拉. 沉浸式叙事视角下的中华文化数字化传承与传播[J]. 包装工程, 2023, 44(2): 1-68.
XU Y Q, TU L.Digital Inheritance and Transmission of Chinese Culture from the Perspective of Immersive Narrative[J]. Packaging Engineering, 2023, 44(2): 1-68.
[15] 图拉, 关玲. 狭义媒介论视角下的沉浸式媒介叙事研究[J]. 现代传播(中国传媒大学学报), 2022, 44(9): 113-121.
TU L, GUAN L.Research on Immersive Media Narration from the Perspective of Narrow Media Theory[J]. Modern Communication (Journal of Communication University of China), 2022, 44(9): 113-121.
[16] 王思豪, 许结. 圣域的图写: 从《上林赋》到《上林图》[J]. 复旦学报(社会科学版), 2015, 57(5): 94-105.
WANG S H, XU J.Writing and Drawing "Sanctuary": From Shanglin Fu to Shanglin Picture[J]. Fudan Journal (Social Sciences Edition), 2015, 57(5): 94-105.
[17] 李奕成, 陈霆锋, 耿华峥, 等. 秦汉时期京畿“城苑一体” 背景下皇家苑囿的涉水营建[J]. 中国园林, 2024, 40(12): 124-130.
LI Y C, CHEN T F, GENG H Z, et al.The Water-Wading Construction of Yuanyou in the Context of the "Integration of the Capital City and Landscape" during Qin and Han Dynasties[J]. Chinese Landscape Architecture, 2024, 40(12): 124-130.
[18] 张雨. 汉都长安的自然美学考察[J]. 郑州大学学报(哲学社会科学版), 2017, 50(1): 14-18.
ZHANG Y.Investigation on Natural Aesthetics of Chang 'An, the Capital of Han Dynasty[J]. Journal of Zhengzhou University (Philosophy and Social Sciences Edition), 2017, 50(1): 14-18.
[19] 赵天一. 中国古典体象论[J]. 西南大学学报(社会科学版), 2016, 42(2): 138-146.
ZHAO T Y.China's Classical Body Image Theory[J]. Journal of Southwest University (Social Sciences Edition), 2016, 42(2): 138-146.
[20] 周金泰. 陈列帝国:汉上林苑的博物空间及其方术、政治理想[J]. 史学月刊, 2023(2): 37-50.
ZHOU J T.The Empire in Display: The Natural History Space of Shanglin Garden of Han and Its Necromancy and Political Ideal[J]. Journal of Historical Science, 2023(2): 37-50.
[21] 边坤, 韩冬楠, 李思雨, 等. 多模态人机交互设计研究进展[J]. 机械设计, 2024, 41(11): 199-204.
BIAN K, HAN D N, LI S Y, et al.Research Progress of Multimodal Human-Computer Interaction Design[J]. Journal of Machine Design, 2024, 41(11): 199-204.
[22] SIU A F, SINCLAIR M, KOVACS R, et al.Virtual Reality without Vision: A Haptic and Auditory White Cane to Navigate Complex Virtual Worlds[C]//Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems. Honolulu: ACM, 2020: 1-13.
[23] PATNAIK B, BATCH A, ELMQVIST N.Information Olfactation: Harnessing Scent to Convey Data[J]. IEEE Transactions on Visualization and Computer Graphics, 2018, 25(1): 726-736.
[24] 董昕. “散点透视” 与华语电影的民族化影像风格[J]. 文艺争鸣, 2024(6): 204-208.
DONG X."Scattered Perspective" and the Nationalized Image Style of Chinese Films[J]. Literary and Artistic Contention, 2024(6): 204-208.
[25] 冯乃恩. 博物馆数字化建设理念与实践综述——以数字故宫社区为例[J]. 故宫博物院院刊, 2017(1): 108-162.
FENG N E.The Idea and Practice for Digitalized Museum Construction: The Case Study of the Palace Museum Digital Community[J]. Palace Museum Journal, 2017(1): 108-162.
[26] 方丽瑜. 博物馆在线藏品数据库的关键质量框架设计——以故宫博物院数字文物库为例[J]. 故宫博物院院刊, 2024(5): 13-140.
FANG L Y.On-Line Museum Collection Database Key- Quality Framework Design with the Case Study of the Palace Museum Digitalized Cultural Relics Library[J]. Palace Museum Journal, 2024(5): 13-140.