https://github.com/JiuhaiChen/BLIP3o\n","updatedAt":"2025-05-15T01:53:18.023Z","author":{"_id":"6393847e3e30234ae798b7be","avatarUrl":"/avatars/daeb8c37dff4432d837a69b87c196521.svg","fullname":"JiuhaiChen","name":"jiuhai","type":"user","isPro":true,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":30}},"numEdits":1,"identifiedLanguage":{"language":"en","probability":0.6811232566833496},"editors":["jiuhai"],"editorAvatarUrls":["/avatars/daeb8c37dff4432d837a69b87c196521.svg"],"reactions":[],"isReport":false}},{"id":"6826530a378176562ebac59e","author":{"_id":"6813ee19c9b224a738fea856","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/g1uPHIKEgWe1ftHGHbo_U.png","fullname":"YJ","name":"yjh415","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false},"createdAt":"2025-05-15T20:48:10.000Z","type":"comment","data":{"edited":false,"hidden":false,"latest":{"raw":"an audio overview for learning on the go: https://youtu.be/z5dMx-Azpxs","html":"
an audio overview for learning on the go: https://youtu.be/z5dMx-Azpxs
\n","updatedAt":"2025-05-15T20:48:10.191Z","author":{"_id":"6813ee19c9b224a738fea856","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/g1uPHIKEgWe1ftHGHbo_U.png","fullname":"YJ","name":"yjh415","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.3325701951980591},"editors":["yjh415"],"editorAvatarUrls":["https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/g1uPHIKEgWe1ftHGHbo_U.png"],"reactions":[],"isReport":false}},{"id":"68269650f032d814753f7377","author":{"_id":"63d3e0e8ff1384ce6c5dd17d","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1674830754237-63d3e0e8ff1384ce6c5dd17d.jpeg","fullname":"Librarian Bot (Bot)","name":"librarian-bot","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":264},"createdAt":"2025-05-16T01:35:12.000Z","type":"comment","data":{"edited":false,"hidden":false,"latest":{"raw":"This is an automated message from the [Librarian Bot](https://huggingface.co/librarian-bots). I found the following papers similar to this paper. \n\nThe following papers were recommended by the Semantic Scholar API \n\n* [An Empirical Study of GPT-4o Image Generation Capabilities](https://huggingface.co/papers/2504.05979) (2025)\n* [Harmonizing Visual Representations for Unified Multimodal Understanding and Generation](https://huggingface.co/papers/2503.21979) (2025)\n* [Preliminary Explorations with GPT-4o(mni) Native Image Generation](https://huggingface.co/papers/2505.05501) (2025)\n* [Ming-Lite-Uni: Advancements in Unified Architecture for Natural Multimodal Interaction](https://huggingface.co/papers/2505.02471) (2025)\n* [Nexus-Gen: A Unified Model for Image Understanding, Generation, and Editing](https://huggingface.co/papers/2504.21356) (2025)\n* [Transfer between Modalities with MetaQueries](https://huggingface.co/papers/2504.06256) (2025)\n* [Mogao: An Omni Foundation Model for Interleaved Multi-Modal Generation](https://huggingface.co/papers/2505.05472) (2025)\n\n\n Please give a thumbs up to this comment if you found it helpful!\n\n If you want recommendations for any Paper on Hugging Face checkout [this](https://huggingface.co/spaces/librarian-bots/recommend_similar_papers) Space\n\n You can directly ask Librarian Bot for paper recommendations by tagging it in a comment: `@librarian-bot recommend`","html":"This is an automated message from the Librarian Bot. I found the following papers similar to this paper.
\nThe following papers were recommended by the Semantic Scholar API
\n- \n
- An Empirical Study of GPT-4o Image Generation Capabilities (2025) \n
- Harmonizing Visual Representations for Unified Multimodal Understanding and Generation (2025) \n
- Preliminary Explorations with GPT-4o(mni) Native Image Generation (2025) \n
- Ming-Lite-Uni: Advancements in Unified Architecture for Natural Multimodal Interaction (2025) \n
- Nexus-Gen: A Unified Model for Image Understanding, Generation, and Editing (2025) \n
- Transfer between Modalities with MetaQueries (2025) \n
- Mogao: An Omni Foundation Model for Interleaved Multi-Modal Generation (2025) \n
Please give a thumbs up to this comment if you found it helpful!
\nIf you want recommendations for any Paper on Hugging Face checkout this Space
\n You can directly ask Librarian Bot for paper recommendations by tagging it in a comment: \n\n@librarian-bot\n\t recommend
Papers
arxiv:2505.09568
BLIP3-o: A Family of Fully Open Unified Multimodal Models-Architecture, Training and Dataset
Abstract
A diffusion transformer is used in a unified multimodal model framework to improve image generation while maintaining image understanding capabilities.
AI-generated summary
Unifying image understanding and generation has gained growing attention in recent research on multimodal models. Although design choices for image understanding have been extensively studied, the optimal model architecture and training recipe for a unified framework with image generation remain underexplored. Motivated by the strong potential of autoregressive and diffusion models for high-quality generation and scalability, we conduct a comprehensive study of their use in unified multimodal settings, with emphasis on image representations, modeling objectives, and training strategies. Grounded in these investigations, we introduce a novel approach that employs a diffusion transformer to generate semantically rich CLIP image features, in contrast to conventional VAE-based representations. This design yields both higher training efficiency and improved generative quality. Furthermore, we demonstrate that a sequential pretraining strategy for unified models-first training on image understanding and subsequently on image generation-offers practical advantages by preserving image understanding capability while developing strong image generation ability. Finally, we carefully curate a high-quality instruction-tuning dataset BLIP3o-60k for image generation by prompting GPT-4o with a diverse set of captions covering various scenes, objects, human gestures, and more. Building on our innovative model design, training recipe, and datasets, we develop BLIP3-o, a suite of state-of-the-art unified multimodal models. BLIP3-o achieves superior performance across most of the popular benchmarks spanning both image understanding and generation tasks. To facilitate future research, we fully open-source our models, including code, model weights, training scripts, and pretraining and instruction tuning datasets.
Community
BLIP3-o: A Family of Fully Open Unified Multimodal Models—Architecture, Training and Dataset
Github: https://github.com/JiuhaiChen/BLIP3o
This is an automated message from the Librarian Bot. I found the following papers similar to this paper.
The following papers were recommended by the Semantic Scholar API
- An Empirical Study of GPT-4o Image Generation Capabilities (2025)
- Harmonizing Visual Representations for Unified Multimodal Understanding and Generation (2025)
- Preliminary Explorations with GPT-4o(mni) Native Image Generation (2025)
- Ming-Lite-Uni: Advancements in Unified Architecture for Natural Multimodal Interaction (2025)
- Nexus-Gen: A Unified Model for Image Understanding, Generation, and Editing (2025)
- Transfer between Modalities with MetaQueries (2025)
- Mogao: An Omni Foundation Model for Interleaved Multi-Modal Generation (2025)
Please give a thumbs up to this comment if you found it helpful!
If you want recommendations for any Paper on Hugging Face checkout this Space
You can directly ask Librarian Bot for paper recommendations by tagging it in a comment:
@librarian-bot
recommend
Models citing this paper 0
No model linking this paper
Cite arxiv.org/abs/2505.09568 in a model README.md to link it from this page.
Datasets citing this paper 0
No dataset linking this paper
Cite arxiv.org/abs/2505.09568 in a dataset README.md to link it from this page.