Please give a thumbs up to this comment if you found it helpful!
\n
If you want recommendations for any Paper on Hugging Face checkout this Space
\n
You can directly ask Librarian Bot for paper recommendations by tagging it in a comment: \n\n@librarian-bot\n\t recommend
\n","updatedAt":"2025-04-11T01:34:22.468Z","author":{"_id":"63d3e0e8ff1384ce6c5dd17d","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1674830754237-63d3e0e8ff1384ce6c5dd17d.jpeg","fullname":"Librarian Bot (Bot)","name":"librarian-bot","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":264}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.7306709885597229},"editors":["librarian-bot"],"editorAvatarUrls":["https://cdn-avatars.huggingface.co/v1/production/uploads/1674830754237-63d3e0e8ff1384ce6c5dd17d.jpeg"],"reactions":[],"isReport":false}}],"primaryEmailConfirmed":false,"paper":{"id":"2504.07089","authors":[{"_id":"67f7676d0ab78ef7b16a820f","user":{"_id":"6614fb3d5aed02b298a4b469","avatarUrl":"/avatars/d0ddb4f989ad1a3f24128cc843347bde.svg","isPro":false,"fullname":"yiting lu","user":"yeeeeeyy","type":"user"},"name":"Yiting Lu","status":"admin_assigned","statusLastChangedAt":"2025-04-10T07:58:03.992Z","hidden":false},{"_id":"67f7676d0ab78ef7b16a8210","user":{"_id":"64a3d1ddb3239f3e3892b24b","avatarUrl":"/avatars/7ce585f5fc1d077fb1d70cc18c4da2c1.svg","isPro":false,"fullname":"Jiakang Yuan","user":"JiakangYuan","type":"user"},"name":"Jiakang Yuan","status":"admin_assigned","statusLastChangedAt":"2025-04-10T08:08:31.119Z","hidden":false},{"_id":"67f7676d0ab78ef7b16a8211","user":{"_id":"6285a9133ab6642179158944","avatarUrl":"/avatars/6e10fa07c94141fcdbe0cab02bb731ca.svg","isPro":false,"fullname":"Zhen Li","user":"Paper99","type":"user"},"name":"Zhen Li","status":"claimed_verified","statusLastChangedAt":"2025-04-10T13:23:24.319Z","hidden":false},{"_id":"67f7676d0ab78ef7b16a8212","user":{"_id":"62c66504031996c36c86976a","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/62c66504031996c36c86976a/wIq0YJhkWnEhlzsh-TGYO.png","isPro":false,"fullname":"steve z","user":"stzhao","type":"user"},"name":"Shitian Zhao","status":"claimed_verified","statusLastChangedAt":"2025-04-11T08:23:49.529Z","hidden":false},{"_id":"67f7676d0ab78ef7b16a8213","user":{"_id":"66bb136002fd8eb58bc84ffb","avatarUrl":"/avatars/122cb8f59c502392768099b3c2afe043.svg","isPro":false,"fullname":"qinqi","user":"Dakerqi","type":"user"},"name":"Qi Qin","status":"claimed_verified","statusLastChangedAt":"2025-04-10T08:06:06.570Z","hidden":false},{"_id":"67f7676d0ab78ef7b16a8214","user":{"_id":"66aba287b0f0b7411f511a47","avatarUrl":"/avatars/1450f182c38e80066ae5ea5df4fa218f.svg","isPro":false,"fullname":"Xinyue Li","user":"Xxxy13","type":"user"},"name":"Xinyue Li","status":"claimed_verified","statusLastChangedAt":"2025-04-11T07:25:55.970Z","hidden":false},{"_id":"67f7676d0ab78ef7b16a8215","user":{"_id":"6358a167f56b03ec9147074d","avatarUrl":"/avatars/e54ea7bf0c240cf76d538296efb3976c.svg","isPro":false,"fullname":"Le Zhuo","user":"JackyZhuo","type":"user"},"name":"Le Zhuo","status":"claimed_verified","statusLastChangedAt":"2025-09-13T15:05:21.489Z","hidden":false},{"_id":"67f7676d0ab78ef7b16a8216","user":{"_id":"64a7c43ae940d769194055df","avatarUrl":"/avatars/441ccadd62e039fb8cb112f138ed917d.svg","isPro":false,"fullname":"Licheng Wen","user":"Wayne-lc","type":"user"},"name":"Licheng Wen","status":"admin_assigned","statusLastChangedAt":"2025-04-10T08:09:07.684Z","hidden":false},{"_id":"67f7676d0ab78ef7b16a8217","user":{"_id":"646f1bef075e11ca78da3bb7","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/646f1bef075e11ca78da3bb7/gNS-ikyZXYeMrf4a7HTQE.jpeg","isPro":false,"fullname":"Dongyang Liu (Chris Liu)","user":"Cxxs","type":"user"},"name":"Dongyang Liu","status":"admin_assigned","statusLastChangedAt":"2025-04-10T08:09:21.398Z","hidden":false},{"_id":"67f7676d0ab78ef7b16a8218","name":"Yuewen Cao","hidden":false},{"_id":"67f7676d0ab78ef7b16a8219","user":{"_id":"65b88b92e0bde92c176a888a","avatarUrl":"/avatars/fc1cb54328ca93860e97fc73a3c1eb2f.svg","isPro":false,"fullname":"Xiangchao Yan","user":"yxc97","type":"user"},"name":"Xiangchao Yan","status":"admin_assigned","statusLastChangedAt":"2025-04-10T08:09:35.388Z","hidden":false},{"_id":"67f7676d0ab78ef7b16a821a","name":"Xin Li","hidden":false},{"_id":"67f7676d0ab78ef7b16a821b","user":{"_id":"643df87f7cd64d872cb9fabd","avatarUrl":"/avatars/c53bfabcee08de448dde973915e8b31d.svg","isPro":false,"fullname":"Botian Shi","user":"friskit","type":"user"},"name":"Botian Shi","status":"admin_assigned","statusLastChangedAt":"2025-04-10T08:09:41.754Z","hidden":false},{"_id":"67f7676d0ab78ef7b16a821c","name":"Tao Chen","hidden":false},{"_id":"67f7676d0ab78ef7b16a821d","user":{"_id":"66d963e52e82d53d3b81031b","avatarUrl":"/avatars/302dbffc033ff47813a2435a2cec02f1.svg","isPro":false,"fullname":"Zhibo Chen","user":"winhelp","type":"user"},"name":"Zhibo Chen","status":"admin_assigned","statusLastChangedAt":"2025-04-10T08:10:04.682Z","hidden":false},{"_id":"67f7676d0ab78ef7b16a821e","name":"Lei Bai","hidden":false},{"_id":"67f7676d0ab78ef7b16a821f","user":{"_id":"643dfd235aafbdca3a5792c0","avatarUrl":"/avatars/ce8553cf5936012c692e08054ee27937.svg","isPro":false,"fullname":"Bo Zhang","user":"BoZhang","type":"user"},"name":"Bo Zhang","status":"claimed_verified","statusLastChangedAt":"2025-04-10T09:57:56.032Z","hidden":false},{"_id":"67f7676d0ab78ef7b16a8220","name":"Peng Gao","hidden":false}],"publishedAt":"2025-04-09T17:58:58.000Z","submittedOnDailyAt":"2025-04-10T05:22:13.319Z","title":"OmniCaptioner: One Captioner to Rule Them All","submittedOnDailyBy":{"_id":"6614fb3d5aed02b298a4b469","avatarUrl":"/avatars/d0ddb4f989ad1a3f24128cc843347bde.svg","isPro":false,"fullname":"yiting lu","user":"yeeeeeyy","type":"user"},"summary":"We propose OmniCaptioner, a versatile visual captioning framework for\ngenerating fine-grained textual descriptions across a wide variety of visual\ndomains. Unlike prior methods limited to specific image types (e.g., natural\nimages or geometric visuals), our framework provides a unified solution for\ncaptioning natural images, visual text (e.g., posters, UIs, textbooks), and\nstructured visuals (e.g., documents, tables, charts). By converting low-level\npixel information into semantically rich textual representations, our framework\nbridges the gap between visual and textual modalities. Our results highlight\nthree key advantages: (i) Enhanced Visual Reasoning with LLMs, where\nlong-context captions of visual modalities empower LLMs, particularly the\nDeepSeek-R1 series, to reason effectively in multimodal scenarios; (ii)\nImproved Image Generation, where detailed captions improve tasks like\ntext-to-image generation and image transformation; and (iii) Efficient\nSupervised Fine-Tuning (SFT), which enables faster convergence with less data.\nWe believe the versatility and adaptability of OmniCaptioner can offer a new\nperspective for bridging the gap between language and visual modalities.","upvotes":20,"discussionId":"67f767700ab78ef7b16a82d6","ai_summary":"OmniCaptioner generates detailed captions across various visual domains, enhancing visual reasoning with LLMs, improving image generation tasks, and enabling efficient supervised fine-tuning.","ai_keywords":["visual captioning framework","low-level pixel information","semantically rich textual representations","LLMs","DeepSeek-R1","long-context captions","multimodal scenarios","text-to-image generation","image transformation","supervised fine-tuning"]},"canReadDatabase":false,"canManagePapers":false,"canSubmit":false,"hasHfLevelAccess":false,"upvoted":false,"upvoters":[{"_id":"648eb1eb59c4e5c87dc116e0","avatarUrl":"/avatars/c636cea39c2c0937f01398c94ead5dad.svg","isPro":false,"fullname":"fdsqefsgergd","user":"T-representer","type":"user"},{"_id":"6285a9133ab6642179158944","avatarUrl":"/avatars/6e10fa07c94141fcdbe0cab02bb731ca.svg","isPro":false,"fullname":"Zhen Li","user":"Paper99","type":"user"},{"_id":"66bb136002fd8eb58bc84ffb","avatarUrl":"/avatars/122cb8f59c502392768099b3c2afe043.svg","isPro":false,"fullname":"qinqi","user":"Dakerqi","type":"user"},{"_id":"64a3d1ddb3239f3e3892b24b","avatarUrl":"/avatars/7ce585f5fc1d077fb1d70cc18c4da2c1.svg","isPro":false,"fullname":"Jiakang Yuan","user":"JiakangYuan","type":"user"},{"_id":"643dfd235aafbdca3a5792c0","avatarUrl":"/avatars/ce8553cf5936012c692e08054ee27937.svg","isPro":false,"fullname":"Bo Zhang","user":"BoZhang","type":"user"},{"_id":"635964636a61954080850e1d","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/635964636a61954080850e1d/0bfExuDTrHTtm8c-40cDM.png","isPro":false,"fullname":"William Lamkin","user":"phanes","type":"user"},{"_id":"620783f24e28382272337ba4","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/620783f24e28382272337ba4/zkUveQPNiDfYjgGhuFErj.jpeg","isPro":false,"fullname":"GuoLiangTang","user":"Tommy930","type":"user"},{"_id":"6039478ab3ecf716b1a5fd4d","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/6039478ab3ecf716b1a5fd4d/_Thy4E7taiSYBLKxEKJbT.jpeg","isPro":true,"fullname":"taesiri","user":"taesiri","type":"user"},{"_id":"6595abdaf96445366607070e","avatarUrl":"/avatars/0b95d4917d92b7e15d11bae5935ddb93.svg","isPro":true,"fullname":"Sam Flin","user":"sflindrs","type":"user"},{"_id":"64c1c77c245c55a21c6f5a13","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/64c1c77c245c55a21c6f5a13/d9zlSksf3TxWpBbb-r0fd.jpeg","isPro":false,"fullname":"Reza Sayar","user":"Reza2kn","type":"user"},{"_id":"644616965691ca69b0e02e79","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/F2uQXO9SkQwHU6benSwQB.jpeg","isPro":false,"fullname":"Juncheng Yan","user":"JonsonYan","type":"user"},{"_id":"66aba287b0f0b7411f511a47","avatarUrl":"/avatars/1450f182c38e80066ae5ea5df4fa218f.svg","isPro":false,"fullname":"Xinyue Li","user":"Xxxy13","type":"user"}],"acceptLanguages":["*"],"dailyPaperRank":0}">
OmniCaptioner generates detailed captions across various visual domains, enhancing visual reasoning with LLMs, improving image generation tasks, and enabling efficient supervised fine-tuning.
AI-generated summary
We propose OmniCaptioner, a versatile visual captioning framework for
generating fine-grained textual descriptions across a wide variety of visual
domains. Unlike prior methods limited to specific image types (e.g., natural
images or geometric visuals), our framework provides a unified solution for
captioning natural images, visual text (e.g., posters, UIs, textbooks), and
structured visuals (e.g., documents, tables, charts). By converting low-level
pixel information into semantically rich textual representations, our framework
bridges the gap between visual and textual modalities. Our results highlight
three key advantages: (i) Enhanced Visual Reasoning with LLMs, where
long-context captions of visual modalities empower LLMs, particularly the
DeepSeek-R1 series, to reason effectively in multimodal scenarios; (ii)
Improved Image Generation, where detailed captions improve tasks like
text-to-image generation and image transformation; and (iii) Efficient
Supervised Fine-Tuning (SFT), which enables faster convergence with less data.
We believe the versatility and adaptability of OmniCaptioner can offer a new
perspective for bridging the gap between language and visual modalities.
We propose OMNICAPTIONER, a versatile visual captioning framework for generating fine-grained textual descriptions across a wide variety of visual domains. Unlike prior methods limited to specific image types (e.g., natural images or geometric visuals), our framework provides a unified solution for captioning natural images, visual text (e.g., posters, UIs, textbooks), and structured visuals (e.g., documents, tables, charts). By converting low-level pixel information into semantically rich textual representations, our framework bridges the gap between visual and textual modalities. Our results highlight three key advantages: (i) Enhanced Visual Reasoning with LLMs, where long-context captions of visual modalities empower LLMs, particularly the DeepSeek-R1 series, to reason effectively in multimodal scenarios; (ii) Improved Image Generation, where detailed captions improve tasks like text-to-image generation and image transformation; and (iii) Efficient Supervised Fine-Tuning (SFT), which enables faster convergence with less data. We believe the versatility and adaptability of OMNICAPTIONER can offer a new perspective for bridging the gap between language and visual modalities.