ShanghaiTech University Knowledge Management System
| Instruction-ViT: Multi-Modal Prompts for Instruction Learning in ViT | |
| 2023-04-29 | |
| 状态 | 已发表 |
| 摘要 | Prompts have been proven to play a crucial role in large language models, and in recent years, vision models have also been using prompts to improve scalability for multiple downstream tasks. In this paper, we focus on adapting prompt design based on instruction tuning into a visual transformer model for image classification which we called Instruction-ViT. The key idea is to implement multi-modal prompts (text or image prompt) related to category information to guide the fine-tuning of the model. Based on the experiments of several image captionining tasks, the performance and domain adaptability were improved. Our work provided an innovative strategy to fuse multi-modal prompts with better performance and faster adaptability for visual classification models. |
| DOI | arXiv:2305.00201 |
| 相关网址 | 查看原文 |
| 出处 | Arxiv |
| WOS记录号 | PPRN:66504933 |
| WOS类目 | Computer Science, Software Engineering |
| 文献类型 | 预印本 |
| 条目标识符 | https://kms.shanghaitech.edu.cn/handle/2MSLDSTB/348137 |
| 专题 | 生物医学工程学院 |
| 作者单位 | 1.Univ Elect Sci & Technol China, Sch Life Sci & Technol, Chengdu 611731, Peoples R China 2.ShanghaiTech Univ, Sch Biomed Engn, Shanghai 201210, Peoples R China 3.Shanghai United Imaging Intelligence Co Ltd, Shanghai 200230, Peoples R China 4.Shanghai Clin Res & Trial Ctr, Shanghai 201210, Peoples R China 5.Univ Texas Arlington, Dept Comp Sci & Engn, Arlington, TX 76019, USA 6.Univ Georgia, Sch Comp, Athens, GA 30602, USA 7.Univ Elect Sci & Technol China, Glasgow Coll, Chengdu 611731, Peoples R China 8.Northwestern Polytech Univ, Sch Automat, Xian 710072, Peoples R China 9.Chinese Univ Hong Kong, Dept Elect Engn, Hong Kong, Peoples R China 10.MayoClin, Dept Radiot Oncol, Phoenix, AZ, USA 11.Massachusetts Gen Hosp, Dept Radiol, Boston, MA 02115, USA 12.Harvard Med Sch, Boston, MA 02115, USA |
| 推荐引用方式 GB/T 7714 | Xiao, Zhenxiang,Chen, Yuzhong,Zhang, Lu,et al. Instruction-ViT: Multi-Modal Prompts for Instruction Learning in ViT. 2023. |
| 条目包含的文件 | ||||||
| 文件名称/大小 | 文献类型 | 版本类型 | 开放类型 | 使用许可 | ||
修改评论
除非特别说明，本系统中所有内容都受版权保护，并保留所有权利。