ShanghaiTech University Knowledge Management System
| Language and Sketching: An LLM-driven Interactive Multimodal Multitask Robot Navigation Framework | |
| 2024 | |
| 会议录名称 | PROCEEDINGS - IEEE INTERNATIONAL CONFERENCE ON ROBOTICS AND AUTOMATION |
| ISSN | 1050-4729 |
| 页码 | 1019-1025 |
| 发表状态 | 已发表 |
| DOI | 10.1109/ICRA57147.2024.10611462 |
| 摘要 | The socially-aware navigation system has evolved to adeptly avoid various obstacles while performing multiple tasks, such as point-to-point navigation, human-following, and-guiding. However, a prominent gap persists: in Human-Robot Interaction (HRI), the procedure of communicating commands to robots demands intricate mathematical formulations. Furthermore, the transition between tasks does not quite possess the intuitive control and user-centric interactivity that one would desire. In this work, we propose an LLM-driven interactive multimodal multitask robot navigation framework, termed LIM2N, to solve the above new challenge in the navigation field. We achieve this by first introducing a multimodal interaction framework where language and hand-drawn inputs can serve as navigation constraints and control objectives. Next, a reinforcement learning agent is built to handle multiple tasks with the received information. Crucially, LIM2N creates smooth cooperation among the reasoning of multimodal input, multitask planning, and adaptation and processing of the intelligent sensing modules in the complicated system. Detailed experiments are conducted in both simulation and the real world demonstrating that LIM2N has solid user needs understanding, alongside an enhanced interactive experience. © 2024 IEEE. |
| 会议录编者/会议主办者 | Beijing NOKOV Science and Technology Co., Ltd. ; et al. ; Kawasaki Heavy Industries, Ltd. ; Kuka AG ; Schunk SE and Co. KG ; ShangHai CHINGMU Tech Ltd |
| 关键词 | Adversarial machine learning Industrial robots Intelligent robots Microrobots Multipurpose robots Nanorobots Reinforcement learning Robot programming Human following Humans-robot interactions Intuitive controls Mathematical formulation Multi-modal Multiple tasks Point-to-point navigation Robot navigation frameworks Sketchings User-centric |
| 会议名称 | 2024 IEEE International Conference on Robotics and Automation, ICRA 2024 |
| 会议地点 | 1-1-1, Minato Mirai, Nishi-ku, Yokohama, Japan |
| 会议日期 | May 13, 2024 - May 17, 2024 |
| URL | 查看原文 |
| 收录类别 | EI |
| 语种 | 英语 |
| 出版者 | Institute of Electrical and Electronics Engineers Inc. |
| EI入藏号 | 20243516962255 |
| EI主题词 | Human robot interaction |
| EI分类号 | 101.6.1 ; 1101.2 ; 1106.1 ; 1107 ; 731.5 Robotics ; 731.6 Robot Applications |
| 原始文献类型 | Conference article (CA) |
| 来源库 | IEEE |
| 引用统计 | 正在获取...
|
| 文献类型 | 会议论文 |
| 条目标识符 | https://kms.shanghaitech.edu.cn/handle/2MSLDSTB/421458 |
| 专题 | 信息科学与技术学院_博士生 信息科学与技术学院_硕士生 创意与艺术学院_PI研究组(P)_田政组 |
| 通讯作者 | Sun, Fanglei |
| 作者单位 | 1.ShanghaiTech University, Shanghai, China; 2.Harbin Institute of Technology, Harbin, China; 3.University of Shanghai for Science and Technology, Shanghai, China; 4.University of Manchester, Manchester, United Kingdom; 5.University College London, London, United Kingdom |
| 第一作者单位 | 上海科技大学 |
| 第一作者的第一单位 | 上海科技大学 |
| 推荐引用方式 GB/T 7714 | Zu, Weiqin,Song, Wenbin,Chen, Ruiqing,et al. Language and Sketching: An LLM-driven Interactive Multimodal Multitask Robot Navigation Framework[C]//Beijing NOKOV Science and Technology Co., Ltd., et al., Kawasaki Heavy Industries, Ltd., Kuka AG, Schunk SE and Co. KG, ShangHai CHINGMU Tech Ltd:Institute of Electrical and Electronics Engineers Inc.,2024:1019-1025. |
| 条目包含的文件 | 下载所有文件 | |||||
| 文件名称/大小 | 文献类型 | 版本类型 | 开放类型 | 使用许可 | ||
修改评论
除非特别说明，本系统中所有内容都受版权保护，并保留所有权利。