上海科技大学知识管理系统

Multilingual multimodal pre-training has garnered significant attention, but it faces challenges due to the substantial need for diverse multilingual text-image data, especially for minor languages. This article introduces UniGen, a unified strategy for efficient multilingual multimodal pre-training inspired by internet data distribution observations. Leveraging the richer availability and higher quality of multilingual text-English text and English text-image data, UniGen aligns the latent space of multilingual text with visual information to a unified semantic space. This alignment, with English as a reference, proves effective in enhancing cross-modal understanding. UniGen reduces reliance on multilingual text-image data, surpassing comparable models in multilingual multimodal benchmark IGLUE by a notable 7%. Notably, UniGen is the first multilingual multimodal model to unify all pre-training tasks within a generative pre-training framework. © 2024 ACM.