https://huggingface.co/HuggingFaceM4/Idefics3-8B-Llama3

\n","updatedAt":"2024-08-26T08:06:00.465Z","author":{"_id":"6177322d37f32ecb1e2d4cdf","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1635201569275-noauth.jpeg","fullname":"Hugo Laurençon","name":"HugoLaurencon","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":159}},"numEdits":1,"identifiedLanguage":{"language":"fr","probability":0.32275810837745667},"editors":["HugoLaurencon"],"editorAvatarUrls":["https://cdn-avatars.huggingface.co/v1/production/uploads/1635201569275-noauth.jpeg"],"reactions":[],"isReport":false}},{"id":"66cc2d33a36a6866174bff05","author":{"_id":"6453ef027d59f53d83f4e5ec","avatarUrl":"/avatars/f6d0689e2b474ddc4032a9782e2f35dc.svg","fullname":"FLORIAN EL-KHAYER","name":"felkhayer","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false},"createdAt":"2024-08-26T07:22:27.000Z","type":"comment","data":{"edited":true,"hidden":true,"hiddenBy":"","latest":{"raw":"This comment has been hidden","html":"This comment has been hidden","updatedAt":"2024-08-26T07:24:35.173Z","author":{"_id":"6453ef027d59f53d83f4e5ec","avatarUrl":"/avatars/f6d0689e2b474ddc4032a9782e2f35dc.svg","fullname":"FLORIAN EL-KHAYER","name":"felkhayer","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false}},"numEdits":0,"editors":[],"editorAvatarUrls":[],"reactions":[]}},{"id":"66cd2cb3da96c1bcdaa425a7","author":{"_id":"63d3e0e8ff1384ce6c5dd17d","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1674830754237-63d3e0e8ff1384ce6c5dd17d.jpeg","fullname":"Librarian Bot (Bot)","name":"librarian-bot","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":264},"createdAt":"2024-08-27T01:32:35.000Z","type":"comment","data":{"edited":false,"hidden":false,"latest":{"raw":"This is an automated message from the [Librarian Bot](https://huggingface.co/librarian-bots). I found the following papers similar to this paper. \n\nThe following papers were recommended by the Semantic Scholar API \n\n* [A Single Transformer for Scalable Vision-Language Modeling](https://huggingface.co/papers/2407.06438) (2024)\n* [OmChat: A Recipe to Train Multimodal Language Models with Strong Long Context and Video Understanding](https://huggingface.co/papers/2407.04923) (2024)\n* [mPLUG-Owl3: Towards Long Image-Sequence Understanding in Multi-Modal Large Language Models](https://huggingface.co/papers/2408.04840) (2024)\n* [EVLM: An Efficient Vision-Language Model for Visual Understanding](https://huggingface.co/papers/2407.14177) (2024)\n* [SynthVLM: High-Efficiency and High-Quality Synthetic Data for Vision Language Models](https://huggingface.co/papers/2407.20756) (2024)\n\n\n Please give a thumbs up to this comment if you found it helpful!\n\n If you want recommendations for any Paper on Hugging Face checkout [this](https://huggingface.co/spaces/librarian-bots/recommend_similar_papers) Space\n\n You can directly ask Librarian Bot for paper recommendations by tagging it in a comment: `@librarian-bot recommend`","html":"

This is an automated message from the Librarian Bot. I found the following papers similar to this paper.

The following papers were recommended by the Semantic Scholar API

Please give a thumbs up to this comment if you found it helpful!

If you want recommendations for any Paper on Hugging Face checkout this Space

You can directly ask Librarian Bot for paper recommendations by tagging it in a comment: \n\n@librarian-bot\n\t recommend

\n","updatedAt":"2024-08-27T01:32:35.324Z","author":{"_id":"63d3e0e8ff1384ce6c5dd17d","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1674830754237-63d3e0e8ff1384ce6c5dd17d.jpeg","fullname":"Librarian Bot (Bot)","name":"librarian-bot","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":264}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.7295231223106384},"editors":["librarian-bot"],"editorAvatarUrls":["https://cdn-avatars.huggingface.co/v1/production/uploads/1674830754237-63d3e0e8ff1384ce6c5dd17d.jpeg"],"reactions":[],"isReport":false}},{"id":"66d145e21dbd780574bd2182","author":{"_id":"610a70f35a40a8bfebfbf09b","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1659922312540-610a70f35a40a8bfebfbf09b.jpeg","fullname":"Daniel Bourke","name":"mrdbourke","type":"user","isPro":true,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":155},"createdAt":"2024-08-30T04:09:06.000Z","type":"comment","data":{"edited":false,"hidden":false,"latest":{"raw":"This was an absolute joy to read! Thank you for the excellent model/paper.","html":"

This was an absolute joy to read! Thank you for the excellent model/paper.

\n","updatedAt":"2024-08-30T04:09:06.533Z","author":{"_id":"610a70f35a40a8bfebfbf09b","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1659922312540-610a70f35a40a8bfebfbf09b.jpeg","fullname":"Daniel Bourke","name":"mrdbourke","type":"user","isPro":true,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":155}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.9487309455871582},"editors":["mrdbourke"],"editorAvatarUrls":["https://cdn-avatars.huggingface.co/v1/production/uploads/1659922312540-610a70f35a40a8bfebfbf09b.jpeg"],"reactions":[],"isReport":false},"replies":[{"id":"66d48169b26010e5719e9281","author":{"_id":"6177322d37f32ecb1e2d4cdf","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1635201569275-noauth.jpeg","fullname":"Hugo Laurençon","name":"HugoLaurencon","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":159},"createdAt":"2024-09-01T14:59:53.000Z","type":"comment","data":{"edited":false,"hidden":false,"latest":{"raw":"Thanks!","html":"

Thanks!

\n","updatedAt":"2024-09-01T14:59:53.799Z","author":{"_id":"6177322d37f32ecb1e2d4cdf","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1635201569275-noauth.jpeg","fullname":"Hugo Laurençon","name":"HugoLaurencon","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":159}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.7788880467414856},"editors":["HugoLaurencon"],"editorAvatarUrls":["https://cdn-avatars.huggingface.co/v1/production/uploads/1635201569275-noauth.jpeg"],"reactions":[{"reaction":"🚀","users":["muhtasham","HEART77"],"count":2}],"isReport":false,"parentCommentId":"66d145e21dbd780574bd2182"}}]}],"primaryEmailConfirmed":false,"paper":{"id":"2408.12637","authors":[{"_id":"66cc04c301fbc62ecda3f5c2","user":{"_id":"6177322d37f32ecb1e2d4cdf","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1635201569275-noauth.jpeg","isPro":false,"fullname":"Hugo Laurençon","user":"HugoLaurencon","type":"user"},"name":"Hugo Laurençon","status":"claimed_verified","statusLastChangedAt":"2024-08-28T15:57:23.421Z","hidden":false},{"_id":"66cc04c301fbc62ecda3f5c3","user":{"_id":"65d66b494bbd0d92b641cdbb","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/65d66b494bbd0d92b641cdbb/6-7dm7B-JxcoS1QlCPdMN.jpeg","isPro":false,"fullname":"Andres Marafioti","user":"andito","type":"user"},"name":"Andrés Marafioti","status":"claimed_verified","statusLastChangedAt":"2024-08-27T09:22:59.164Z","hidden":false},{"_id":"66cc04c301fbc62ecda3f5c4","name":"Victor Sanh","hidden":false},{"_id":"66cc04c301fbc62ecda3f5c5","user":{"_id":"6244866a456803e9500d0f6a","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1652185658647-6244866a456803e9500d0f6a.jpeg","isPro":false,"fullname":"Leo Tronchon","user":"Leyo","type":"user"},"name":"Léo Tronchon","status":"admin_assigned","statusLastChangedAt":"2024-08-30T13:09:27.214Z","hidden":false}],"publishedAt":"2024-08-22T17:47:24.000Z","submittedOnDailyAt":"2024-08-26T03:00:48.374Z","title":"Building and better understanding vision-language models: insights and\n future directions","submittedOnDailyBy":{"_id":"6177322d37f32ecb1e2d4cdf","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1635201569275-noauth.jpeg","isPro":false,"fullname":"Hugo Laurençon","user":"HugoLaurencon","type":"user"},"summary":"The field of vision-language models (VLMs), which take images and texts as\ninputs and output texts, is rapidly evolving and has yet to reach consensus on\nseveral key aspects of the development pipeline, including data, architecture,\nand training methods. This paper can be seen as a tutorial for building a VLM.\nWe begin by providing a comprehensive overview of the current state-of-the-art\napproaches, highlighting the strengths and weaknesses of each, addressing the\nmajor challenges in the field, and suggesting promising research directions for\nunderexplored areas. We then walk through the practical steps to build\nIdefics3-8B, a powerful VLM that significantly outperforms its predecessor\nIdefics2-8B, while being trained efficiently, exclusively on open datasets, and\nusing a straightforward pipeline. These steps include the creation of Docmatix,\na dataset for improving document understanding capabilities, which is 240 times\nlarger than previously available datasets. We release the model along with the\ndatasets created for its training.","upvotes":133,"discussionId":"66cc04c401fbc62ecda3f604","ai_summary":"A comprehensive tutorial on building vision-language models, detailing the development of Idefics3-8B with an improved dataset enhancing document understanding.","ai_keywords":["vision-language models","VLMs","state-of-the-art","Docmatix","document understanding"]},"canReadDatabase":false,"canManagePapers":false,"canSubmit":false,"hasHfLevelAccess":false,"upvoted":false,"upvoters":[{"_id":"620783f24e28382272337ba4","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/620783f24e28382272337ba4/zkUveQPNiDfYjgGhuFErj.jpeg","isPro":false,"fullname":"GuoLiangTang","user":"Tommy930","type":"user"},{"_id":"64747f7e33192631bacd8831","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/64747f7e33192631bacd8831/dstkZJ4sHJSeqLesV5cOC.jpeg","isPro":false,"fullname":"Taufiq Dwi Purnomo","user":"taufiqdp","type":"user"},{"_id":"62deb6c3520a9fae78bb9bc3","avatarUrl":"/avatars/5d75fffa9bad36d20adb8f47141d1f0b.svg","isPro":false,"fullname":"Literate Goggles","user":"literate-goggles","type":"user"},{"_id":"65d66b494bbd0d92b641cdbb","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/65d66b494bbd0d92b641cdbb/6-7dm7B-JxcoS1QlCPdMN.jpeg","isPro":false,"fullname":"Andres Marafioti","user":"andito","type":"user"},{"_id":"64f955c582673b2a07fbf0ad","avatarUrl":"/avatars/1c98c8be61f6580c1e4ee698fa5c0716.svg","isPro":false,"fullname":"hongyu","user":"learn12138","type":"user"},{"_id":"5e48005437cb5b49818287a5","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/5e48005437cb5b49818287a5/4uCXGGui-9QifAT4qelxU.png","isPro":false,"fullname":"Leandro von Werra","user":"lvwerra","type":"user"},{"_id":"602e6dee60e3dd96631c906e","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1613655355830-noauth.png","isPro":false,"fullname":"Anton Lozhkov","user":"anton-l","type":"user"},{"_id":"61ed0ff29539bc0a3bbc89f4","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/61ed0ff29539bc0a3bbc89f4/iYWK7GParA7Ke5F6q132W.jpeg","isPro":false,"fullname":"Miquel Farré","user":"mfarre","type":"user"},{"_id":"5f0c746619cb630495b814fd","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1594651707950-noauth.jpeg","isPro":true,"fullname":"Lewis Tunstall","user":"lewtun","type":"user"},{"_id":"6632d7e22c4f4bfc3f6a05c2","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/6632d7e22c4f4bfc3f6a05c2/TCDlrb-O5aormNjSX-tyE.png","isPro":false,"fullname":"Mohamed Mekkouri","user":"medmekk","type":"user"},{"_id":"640e21ef3c82bd463ee5a76d","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/640e21ef3c82bd463ee5a76d/nVR1DFPAsiLw6Boys28Rb.jpeg","isPro":false,"fullname":"Dana Aubakirova","user":"danaaubakirova","type":"user"},{"_id":"6032802e1f993496bc14d9e3","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/6032802e1f993496bc14d9e3/w6hr-DEQot4VVkoyRIBiy.png","isPro":false,"fullname":"Omar Sanseviero","user":"osanseviero","type":"user"}],"acceptLanguages":["*"],"dailyPaperRank":1}">

Papers

arxiv:2408.12637

Building and better understanding vision-language models: insights and future directions

Published on Aug 22, 2024

· Submitted by

Hugo Laurençon on Aug 26, 2024

#1 Paper of the day

Upvote

133

Authors:

Hugo Laurençon ,

Andrés Marafioti ,

Léo Tronchon

Abstract

A comprehensive tutorial on building vision-language models, detailing the development of Idefics3-8B with an improved dataset enhancing document understanding.

AI-generated summary

The field of vision-language models (VLMs), which take images and texts as inputs and output texts, is rapidly evolving and has yet to reach consensus on several key aspects of the development pipeline, including data, architecture, and training methods. This paper can be seen as a tutorial for building a VLM. We begin by providing a comprehensive overview of the current state-of-the-art approaches, highlighting the strengths and weaknesses of each, addressing the major challenges in the field, and suggesting promising research directions for underexplored areas. We then walk through the practical steps to build Idefics3-8B, a powerful VLM that significantly outperforms its predecessor Idefics2-8B, while being trained efficiently, exclusively on open datasets, and using a straightforward pipeline. These steps include the creation of Docmatix, a dataset for improving document understanding capabilities, which is 240 times larger than previously available datasets. We release the model along with the datasets created for its training.

View arXiv page View PDF Add to collection