https://yueyang1996.github.io/cosyn/

\n","updatedAt":"2025-02-21T03:38:36.412Z","author":{"_id":"60f1abe7544c2adfd699860c","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1674929746905-60f1abe7544c2adfd699860c.jpeg","fullname":"AK","name":"akhaliq","type":"user","isPro":false,"isHf":true,"isHfAdmin":false,"isMod":false,"followerCount":8229}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.26862865686416626},"editors":["akhaliq"],"editorAvatarUrls":["https://cdn-avatars.huggingface.co/v1/production/uploads/1674929746905-60f1abe7544c2adfd699860c.jpeg"],"reactions":[],"isReport":false}},{"id":"67b9299b022d669c606b8cb4","author":{"_id":"63d3e0e8ff1384ce6c5dd17d","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1674830754237-63d3e0e8ff1384ce6c5dd17d.jpeg","fullname":"Librarian Bot (Bot)","name":"librarian-bot","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":264},"createdAt":"2025-02-22T01:34:19.000Z","type":"comment","data":{"edited":false,"hidden":false,"latest":{"raw":"This is an automated message from the [Librarian Bot](https://huggingface.co/librarian-bots). I found the following papers similar to this paper. \n\nThe following papers were recommended by the Semantic Scholar API \n\n* [SMIR: Efficient Synthetic Data Pipeline To Improve Multi-Image Reasoning](https://huggingface.co/papers/2501.03675) (2025)\n* [Granite Vision: a lightweight, open-source multimodal model for enterprise Intelligence](https://huggingface.co/papers/2502.09927) (2025)\n* [MoColl: Agent-Based Specific and General Model Collaboration for Image Captioning](https://huggingface.co/papers/2501.01834) (2025)\n* [MM-GEN: Enhancing Task Performance Through Targeted Multimodal Data Curation](https://huggingface.co/papers/2501.04155) (2025)\n* [Code-Vision: Evaluating Multimodal LLMs Logic Understanding and Code Generation Capabilities](https://huggingface.co/papers/2502.11829) (2025)\n* [Vision-Driven Prompt Optimization for Large Language Models in Multimodal Generative Tasks](https://huggingface.co/papers/2501.02527) (2025)\n* [ImageRef-VL: Enabling Contextual Image Referencing in Vision-Language Models](https://huggingface.co/papers/2501.12418) (2025)\n\n\n Please give a thumbs up to this comment if you found it helpful!\n\n If you want recommendations for any Paper on Hugging Face checkout [this](https://huggingface.co/spaces/librarian-bots/recommend_similar_papers) Space\n\n You can directly ask Librarian Bot for paper recommendations by tagging it in a comment: `@librarian-bot recommend`","html":"

This is an automated message from the Librarian Bot. I found the following papers similar to this paper.

The following papers were recommended by the Semantic Scholar API

Please give a thumbs up to this comment if you found it helpful!

If you want recommendations for any Paper on Hugging Face checkout this Space

You can directly ask Librarian Bot for paper recommendations by tagging it in a comment: \n\n@librarian-bot\n\t recommend

\n","updatedAt":"2025-02-22T01:34:19.694Z","author":{"_id":"63d3e0e8ff1384ce6c5dd17d","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1674830754237-63d3e0e8ff1384ce6c5dd17d.jpeg","fullname":"Librarian Bot (Bot)","name":"librarian-bot","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":264}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.7143558859825134},"editors":["librarian-bot"],"editorAvatarUrls":["https://cdn-avatars.huggingface.co/v1/production/uploads/1674830754237-63d3e0e8ff1384ce6c5dd17d.jpeg"],"reactions":[],"isReport":false}}],"primaryEmailConfirmed":false,"paper":{"id":"2502.14846","authors":[{"_id":"67b7f4f1b15c19d57189fc5e","user":{"_id":"62f6c68904e5e02f82b04690","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/62f6c68904e5e02f82b04690/kK2-PkeAGzAOLhkfajswf.jpeg","isPro":true,"fullname":"Yue Yang","user":"yyupenn","type":"user"},"name":"Yue Yang","status":"claimed_verified","statusLastChangedAt":"2025-02-21T15:06:20.097Z","hidden":false},{"_id":"67b7f4f1b15c19d57189fc5f","name":"Ajay Patel","hidden":false},{"_id":"67b7f4f1b15c19d57189fc60","user":{"_id":"61c388aa727d1257bf3cf58b","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1670871898141-61c388aa727d1257bf3cf58b.jpeg","isPro":false,"fullname":"Matt Deitke","user":"mattdeitke","type":"user"},"name":"Matt Deitke","status":"admin_assigned","statusLastChangedAt":"2025-02-21T15:03:20.545Z","hidden":false},{"_id":"67b7f4f1b15c19d57189fc61","user":{"_id":"62a108242e30aaf94ec283bb","avatarUrl":"/avatars/4da19ae99df6a3fa2d81336eb59cdaa7.svg","isPro":false,"fullname":"Tanmay Gupta","user":"tanmayg","type":"user"},"name":"Tanmay Gupta","status":"claimed_verified","statusLastChangedAt":"2025-06-10T09:31:13.963Z","hidden":false},{"_id":"67b7f4f1b15c19d57189fc62","user":{"_id":"620acbdb3c0931626a7c9297","avatarUrl":"/avatars/f63b1d225ed81e223d3e8876a5c708c4.svg","isPro":false,"fullname":"Luca Weihs","user":"lucaweihs","type":"user"},"name":"Luca Weihs","status":"admin_assigned","statusLastChangedAt":"2025-02-21T15:02:58.922Z","hidden":false},{"_id":"67b7f4f1b15c19d57189fc63","user":{"_id":"6360289588e41d249ecd3e26","avatarUrl":"/avatars/0e84dbd72f07e99967a8d25cda938efe.svg","isPro":false,"fullname":"Andrew Head","user":"ChittyChins","type":"user"},"name":"Andrew Head","status":"admin_assigned","statusLastChangedAt":"2025-02-21T15:02:53.395Z","hidden":false},{"_id":"67b7f4f1b15c19d57189fc64","user":{"_id":"631f42e1b6628770f6efd87a","avatarUrl":"/avatars/0fa93b3513ebca737cce26dfa5611cf1.svg","isPro":false,"fullname":"Mark Yatskar","user":"myatskar","type":"user"},"name":"Mark Yatskar","status":"admin_assigned","statusLastChangedAt":"2025-02-21T15:02:47.575Z","hidden":false},{"_id":"67b7f4f1b15c19d57189fc65","user":{"_id":"6303ce25fc783bfc744216af","avatarUrl":"/avatars/09f5e87c1f56a1b7f6ef9c5037682285.svg","isPro":false,"fullname":"Chris Callison-Burch","user":"CCB","type":"user"},"name":"Chris Callison-Burch","status":"admin_assigned","statusLastChangedAt":"2025-02-21T15:02:41.757Z","hidden":false},{"_id":"67b7f4f1b15c19d57189fc66","user":{"_id":"66429868ab89e3a3a85668b0","avatarUrl":"/avatars/170e0daa454838deee2bf946f7118651.svg","isPro":false,"fullname":"Ranjay Krishna","user":"ranjaykrishna","type":"user"},"name":"Ranjay Krishna","status":"admin_assigned","statusLastChangedAt":"2025-02-21T15:02:35.267Z","hidden":false},{"_id":"67b7f4f1b15c19d57189fc67","name":"Aniruddha Kembhavi","hidden":false},{"_id":"67b7f4f1b15c19d57189fc68","name":"Christopher Clark","hidden":false}],"publishedAt":"2025-02-20T18:55:30.000Z","submittedOnDailyAt":"2025-02-21T01:08:36.406Z","title":"Scaling Text-Rich Image Understanding via Code-Guided Synthetic\n Multimodal Data Generation","submittedOnDailyBy":{"_id":"60f1abe7544c2adfd699860c","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1674929746905-60f1abe7544c2adfd699860c.jpeg","isPro":false,"fullname":"AK","user":"akhaliq","type":"user"},"summary":"Reasoning about images with rich text, such as charts and documents, is a\ncritical application of vision-language models (VLMs). However, VLMs often\nstruggle in these domains due to the scarcity of diverse text-rich\nvision-language data. To address this challenge, we present CoSyn, a framework\nthat leverages the coding capabilities of text-only large language models\n(LLMs) to automatically create synthetic text-rich multimodal data. Given input\ntext describing a target domain (e.g., \"nutrition fact labels\"), CoSyn prompts\nan LLM to generate code (Python, HTML, LaTeX, etc.) for rendering synthetic\nimages. With the underlying code as textual representations of the synthetic\nimages, CoSyn can generate high-quality instruction-tuning data, again relying\non a text-only LLM. Using CoSyn, we constructed a dataset comprising 400K\nimages and 2.7M rows of vision-language instruction-tuning data. Comprehensive\nexperiments on seven benchmarks demonstrate that models trained on our\nsynthetic data achieve state-of-the-art performance among competitive\nopen-source models, including Llama 3.2, and surpass proprietary models such as\nGPT-4V and Gemini 1.5 Flash. Furthermore, CoSyn can produce synthetic pointing\ndata, enabling VLMs to ground information within input images, showcasing its\npotential for developing multimodal agents capable of acting in real-world\nenvironments.","upvotes":14,"discussionId":"67b7f4f2b15c19d57189fc95","projectPage":"https://yueyang1996.github.io/cosyn/","githubRepo":"https://github.com/allenai/pixmo-docs","ai_summary":"CoSyn uses text-only LLMs to generate synthetic text-rich multimodal data, achieving state-of-the-art performance on VLM benchmarks and enabling synthetic pointing data for grounding information within images.","ai_keywords":["vision-language models","LLMs","synthetic text-rich multimodal data","instruction-tuning data","multimodal agents"],"githubStars":137},"canReadDatabase":false,"canManagePapers":false,"canSubmit":false,"hasHfLevelAccess":false,"upvoted":false,"upvoters":[{"_id":"634dffc49b777beec3bc6448","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1670144568552-634dffc49b777beec3bc6448.jpeg","isPro":false,"fullname":"Zhipeng Yang","user":"svjack","type":"user"},{"_id":"6039478ab3ecf716b1a5fd4d","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/6039478ab3ecf716b1a5fd4d/_Thy4E7taiSYBLKxEKJbT.jpeg","isPro":true,"fullname":"taesiri","user":"taesiri","type":"user"},{"_id":"620783f24e28382272337ba4","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/620783f24e28382272337ba4/zkUveQPNiDfYjgGhuFErj.jpeg","isPro":false,"fullname":"GuoLiangTang","user":"Tommy930","type":"user"},{"_id":"66f612b934b8ac9ffa44f084","avatarUrl":"/avatars/6836c122e19c66c90f1673f28b30d7f0.svg","isPro":false,"fullname":"Tang","user":"tommysally","type":"user"},{"_id":"648eb1eb59c4e5c87dc116e0","avatarUrl":"/avatars/c636cea39c2c0937f01398c94ead5dad.svg","isPro":false,"fullname":"fdsqefsgergd","user":"T-representer","type":"user"},{"_id":"5f17f0a0925b9863e28ad517","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/5f17f0a0925b9863e28ad517/fXIY5i9RLsIa1v3CCuVtt.jpeg","isPro":true,"fullname":"Victor Mustar","user":"victor","type":"user"},{"_id":"62f6c68904e5e02f82b04690","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/62f6c68904e5e02f82b04690/kK2-PkeAGzAOLhkfajswf.jpeg","isPro":true,"fullname":"Yue Yang","user":"yyupenn","type":"user"},{"_id":"64fe780614636d417af95e10","avatarUrl":"/avatars/f172b2ad3f66d38205cf9589f3e43585.svg","isPro":false,"fullname":"gerald hewes","user":"gerald29","type":"user"},{"_id":"61c40eeb727d1257bf3cf5ba","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/61c40eeb727d1257bf3cf5ba/hVNbcFjsvwWqWarcGTOdI.jpeg","isPro":false,"fullname":"Ajay Patel","user":"AjayP13","type":"user"},{"_id":"650c8bfb3d3542884da1a845","avatarUrl":"/avatars/863a5deebf2ac6d4faedc4dd368e0561.svg","isPro":false,"fullname":"Adhurim ","user":"Limi07","type":"user"},{"_id":"635964636a61954080850e1d","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/635964636a61954080850e1d/0bfExuDTrHTtm8c-40cDM.png","isPro":false,"fullname":"William Lamkin","user":"phanes","type":"user"},{"_id":"6487239cca30096ea9f52115","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/6487239cca30096ea9f52115/HMte9wjKJgfcxsO-5vb_Q.jpeg","isPro":false,"fullname":"dame rajee","user":"damerajee","type":"user"}],"acceptLanguages":["*"],"dailyPaperRank":0}">

Papers

arxiv:2502.14846

Scaling Text-Rich Image Understanding via Code-Guided Synthetic Multimodal Data Generation

Published on Feb 20

· Submitted by

AK on Feb 21

Upvote

Authors:

Yue Yang ,

Matt Deitke ,

Tanmay Gupta ,

Luca Weihs ,

Andrew Head ,

Mark Yatskar ,

Chris Callison-Burch ,

Ranjay Krishna ,

Abstract

CoSyn uses text-only LLMs to generate synthetic text-rich multimodal data, achieving state-of-the-art performance on VLM benchmarks and enabling synthetic pointing data for grounding information within images.

AI-generated summary

Reasoning about images with rich text, such as charts and documents, is a critical application of vision-language models (VLMs). However, VLMs often struggle in these domains due to the scarcity of diverse text-rich vision-language data. To address this challenge, we present CoSyn, a framework that leverages the coding capabilities of text-only large language models (LLMs) to automatically create synthetic text-rich multimodal data. Given input text describing a target domain (e.g., "nutrition fact labels"), CoSyn prompts an LLM to generate code (Python, HTML, LaTeX, etc.) for rendering synthetic images. With the underlying code as textual representations of the synthetic images, CoSyn can generate high-quality instruction-tuning data, again relying on a text-only LLM. Using CoSyn, we constructed a dataset comprising 400K images and 2.7M rows of vision-language instruction-tuning data. Comprehensive experiments on seven benchmarks demonstrate that models trained on our synthetic data achieve state-of-the-art performance among competitive open-source models, including Llama 3.2, and surpass proprietary models such as GPT-4V and Gemini 1.5 Flash. Furthermore, CoSyn can produce synthetic pointing data, enabling VLMs to ground information within input images, showcasing its potential for developing multimodal agents capable of acting in real-world environments.