$\"video3.gif\"$

💧 Dimple-7B v.s. AR:

Comparable performance
(Sometimes) Faster

🔍 Highlights
Hybrid Training: Combines autoregressive and diffusion training.
Diffusion Decoding: Supports confident decoding, random decoding, maskgit-style decoding, and entropy-based decoding.
Controllable Generation: Enables fine-grained control over format, structure, and length via structure priors.
Autoregressive-like Prefilling: Enhances inference speed using prefilling techniques.

\n","updatedAt":"2025-05-23T02:34:28.764Z","author":{"_id":"635364b3c41f548fe39db945","avatarUrl":"/avatars/ad1916bbfabca0b6651c8eabacc5eba8.svg","fullname":"Runpeng Yu","name":"rp-yu","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":9}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.4754297137260437},"editors":["rp-yu"],"editorAvatarUrls":["/avatars/ad1916bbfabca0b6651c8eabacc5eba8.svg"],"reactions":[],"isReport":false}},{"id":"68312247f477bc6acd320688","author":{"_id":"63d3e0e8ff1384ce6c5dd17d","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1674830754237-63d3e0e8ff1384ce6c5dd17d.jpeg","fullname":"Librarian Bot (Bot)","name":"librarian-bot","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":264},"createdAt":"2025-05-24T01:35:03.000Z","type":"comment","data":{"edited":false,"hidden":false,"latest":{"raw":"This is an automated message from the [Librarian Bot](https://huggingface.co/librarian-bots). I found the following papers similar to this paper. \n\nThe following papers were recommended by the Semantic Scholar API \n\n* [Generative Multimodal Pretraining with Discrete Diffusion Timestep Tokens](https://huggingface.co/papers/2504.14666) (2025)\n* [MMaDA: Multimodal Large Diffusion Language Models](https://huggingface.co/papers/2505.15809) (2025)\n* [dKV-Cache: The Cache for Diffusion Language Models](https://huggingface.co/papers/2505.15781) (2025)\n* [MASSV: Multimodal Adaptation and Self-Data Distillation for Speculative Decoding of Vision-Language Models](https://huggingface.co/papers/2505.10526) (2025)\n* [d1: Scaling Reasoning in Diffusion Large Language Models via Reinforcement Learning](https://huggingface.co/papers/2504.12216) (2025)\n* [Fine-Tuning Visual Autoregressive Models for Subject-Driven Generation](https://huggingface.co/papers/2504.02612) (2025)\n* [Pseudo-Autoregressive Neural Codec Language Models for Efficient Zero-Shot Text-to-Speech Synthesis](https://huggingface.co/papers/2504.10352) (2025)\n\n\n Please give a thumbs up to this comment if you found it helpful!\n\n If you want recommendations for any Paper on Hugging Face checkout [this](https://huggingface.co/spaces/librarian-bots/recommend_similar_papers) Space\n\n You can directly ask Librarian Bot for paper recommendations by tagging it in a comment: `@librarian-bot recommend`","html":"

This is an automated message from the Librarian Bot. I found the following papers similar to this paper.

The following papers were recommended by the Semantic Scholar API

Please give a thumbs up to this comment if you found it helpful!

If you want recommendations for any Paper on Hugging Face checkout this Space

You can directly ask Librarian Bot for paper recommendations by tagging it in a comment: \n\n@librarian-bot\n\t recommend

\n","updatedAt":"2025-05-24T01:35:03.830Z","author":{"_id":"63d3e0e8ff1384ce6c5dd17d","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1674830754237-63d3e0e8ff1384ce6c5dd17d.jpeg","fullname":"Librarian Bot (Bot)","name":"librarian-bot","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":264}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.6397473216056824},"editors":["librarian-bot"],"editorAvatarUrls":["https://cdn-avatars.huggingface.co/v1/production/uploads/1674830754237-63d3e0e8ff1384ce6c5dd17d.jpeg"],"reactions":[],"isReport":false}},{"id":"6836888b79bcae382211711a","author":{"_id":"639daf827270667011153fbc","avatarUrl":"/avatars/0af0894de0744ad83d491e8b59aa65ef.svg","fullname":"Han Yoon","name":"LPX55","type":"user","isPro":true,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":30},"createdAt":"2025-05-28T03:52:43.000Z","type":"comment","data":{"edited":false,"hidden":false,"latest":{"raw":"Awesome work, I was thoroughly impressed 👏","html":"

Awesome work, I was thoroughly impressed 👏

\n","updatedAt":"2025-05-28T03:52:43.817Z","author":{"_id":"639daf827270667011153fbc","avatarUrl":"/avatars/0af0894de0744ad83d491e8b59aa65ef.svg","fullname":"Han Yoon","name":"LPX55","type":"user","isPro":true,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":30}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.9954738020896912},"editors":["LPX55"],"editorAvatarUrls":["/avatars/0af0894de0744ad83d491e8b59aa65ef.svg"],"reactions":[],"isReport":false}},{"id":"683d85ec8c5fef6b407fb2f4","author":{"_id":"64ddca0aae450641ed9334ec","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/_WWGdDaJg1Q8rQLgA2nUF.png","fullname":"Adeer Khan","name":"adeerkhan","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false},"createdAt":"2025-06-02T11:07:24.000Z","type":"comment","data":{"edited":false,"hidden":false,"latest":{"raw":"How did you guys fine-tune the Dream 7B?","html":"

How did you guys fine-tune the Dream 7B?

\n","updatedAt":"2025-06-02T11:07:24.642Z","author":{"_id":"64ddca0aae450641ed9334ec","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/_WWGdDaJg1Q8rQLgA2nUF.png","fullname":"Adeer Khan","name":"adeerkhan","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.9645339846611023},"editors":["adeerkhan"],"editorAvatarUrls":["https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/_WWGdDaJg1Q8rQLgA2nUF.png"],"reactions":[],"isReport":false}}],"primaryEmailConfirmed":false,"paper":{"id":"2505.16990","authors":[{"_id":"682fdd034640a9db4d1cc04d","name":"Runpeng Yu","hidden":false},{"_id":"682fdd034640a9db4d1cc04e","name":"Xinyin Ma","hidden":false},{"_id":"682fdd034640a9db4d1cc04f","name":"Xinchao Wang","hidden":false}],"publishedAt":"2025-05-22T17:55:04.000Z","submittedOnDailyAt":"2025-05-23T01:04:28.755Z","title":"Dimple: Discrete Diffusion Multimodal Large Language Model with Parallel\n Decoding","submittedOnDailyBy":{"_id":"635364b3c41f548fe39db945","avatarUrl":"/avatars/ad1916bbfabca0b6651c8eabacc5eba8.svg","isPro":false,"fullname":"Runpeng Yu","user":"rp-yu","type":"user"},"summary":"In this work, we propose Dimple, the first Discrete Diffusion Multimodal\nLarge Language Model (DMLLM). We observe that training with a purely discrete\ndiffusion approach leads to significant training instability, suboptimal\nperformance, and severe length bias issues. To address these challenges, we\ndesign a novel training paradigm that combines an initial autoregressive phase\nwith a subsequent diffusion phase. This approach yields the Dimple-7B model,\ntrained on the same dataset and using a similar training pipeline as\nLLaVA-NEXT. Dimple-7B ultimately surpasses LLaVA-NEXT in performance by 3.9%,\ndemonstrating that DMLLM can achieve performance comparable to that of\nautoregressive models. To improve inference efficiency, we propose a decoding\nstrategy termed confident decoding, which dynamically adjusts the number of\ntokens generated at each step, significantly reducing the number of generation\niterations. In autoregressive models, the number of forward iterations during\ngeneration equals the response length. With confident decoding, however, the\nnumber of iterations needed by Dimple is even only text{response\nlength}{3}. We also re-implement the prefilling technique in autoregressive\nmodels and demonstrate that it does not significantly impact performance on\nmost benchmark evaluations, while offering a speedup of 1.5x to 7x.\nAdditionally, we explore Dimple's capability to precisely control its response\nusing structure priors. These priors enable structured responses in a manner\ndistinct from instruction-based or chain-of-thought prompting, and allow\nfine-grained control over response format and length, which is difficult to\nachieve in autoregressive models. Overall, this work validates the feasibility\nand advantages of DMLLM and enhances its inference efficiency and\ncontrollability. Code and models are available at\nhttps://github.com/yu-rp/Dimple.","upvotes":21,"discussionId":"682fdd044640a9db4d1cc0a1","githubRepo":"https://github.com/yu-rp/Dimple","ai_summary":"Dimple, a Discrete Diffusion Multimodal Large Language Model, achieves performance comparable to autoregressive models through a hybrid training approach and enhances inference efficiency with confident decoding and structure priors.","ai_keywords":["Discrete Diffusion Multimodal Large Language Model","DMLLM","autoregressive phase","diffusion phase","confident decoding","prefilling technique","structure priors"],"githubStars":100},"canReadDatabase":false,"canManagePapers":false,"canSubmit":false,"hasHfLevelAccess":false,"upvoted":false,"upvoters":[{"_id":"635364b3c41f548fe39db945","avatarUrl":"/avatars/ad1916bbfabca0b6651c8eabacc5eba8.svg","isPro":false,"fullname":"Runpeng Yu","user":"rp-yu","type":"user"},{"_id":"640ebdfefdeaae139086f4d8","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/640ebdfefdeaae139086f4d8/2N94gbHubplYD8njmUTPf.jpeg","isPro":true,"fullname":"Yuanshi","user":"Yuanshi","type":"user"},{"_id":"64828d2b10cd9ffea8b6c14c","avatarUrl":"/avatars/02f3ae1fc567435e41d3892ef44a290a.svg","isPro":false,"fullname":"Deyu Bo","user":"bdy","type":"user"},{"_id":"63492af03caab0136df08d25","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/63492af03caab0136df08d25/-aZzajKl38jT29A-kziCI.jpeg","isPro":false,"fullname":"Shizun Wang","user":"littlepure2333","type":"user"},{"_id":"634cfebc350bcee9bed20a4d","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/634cfebc350bcee9bed20a4d/fN47nN5rhw-HJaFLBZWQy.png","isPro":false,"fullname":"Xingyi Yang","user":"adamdad","type":"user"},{"_id":"620783f24e28382272337ba4","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/620783f24e28382272337ba4/zkUveQPNiDfYjgGhuFErj.jpeg","isPro":false,"fullname":"GuoLiangTang","user":"Tommy930","type":"user"},{"_id":"64396ebc21221ac7411852b3","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/64396ebc21221ac7411852b3/SR0dC8N0bdj9tZFxYPpSf.jpeg","isPro":false,"fullname":"Xinyin Ma","user":"horseee","type":"user"},{"_id":"6684b1a9986286e214df1e03","avatarUrl":"/avatars/515efb62b0ec923ea525a90ea7aa9221.svg","isPro":false,"fullname":"XinyinMa","user":"XinyinHorseee","type":"user"},{"_id":"6321828bcf4e4a0c6ff1661c","avatarUrl":"/avatars/ea733392446e9d562fab1c44f5dc0b1e.svg","isPro":false,"fullname":"ZhihengLiu","user":"Johanan","type":"user"},{"_id":"65811eeaa2284a018e51f1ba","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/dH8UZj6Kk5HJkI1DItCNm.jpeg","isPro":false,"fullname":"Zigeng Chen","user":"Zigeng","type":"user"},{"_id":"6486fb33570a419f41a882e4","avatarUrl":"/avatars/860a42074439a23c629cd23851ae4da6.svg","isPro":false,"fullname":"Ruonan Yu","user":"roseannelexie","type":"user"},{"_id":"5df833bdda6d0311fd3d5403","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/5df833bdda6d0311fd3d5403/62OtGJEQXdOuhV9yCd4HS.png","isPro":false,"fullname":"Weihao Yu","user":"whyu","type":"user"}],"acceptLanguages":["*"],"dailyPaperRank":0}">

Papers

arxiv:2505.16990

Dimple: Discrete Diffusion Multimodal Large Language Model with Parallel Decoding

Published on May 22

· Submitted by

Runpeng Yu on May 23

Upvote

Authors:

Abstract

Dimple, a Discrete Diffusion Multimodal Large Language Model, achieves performance comparable to autoregressive models through a hybrid training approach and enhances inference efficiency with confident decoding and structure priors.

AI-generated summary

In this work, we propose Dimple, the first Discrete Diffusion Multimodal Large Language Model (DMLLM). We observe that training with a purely discrete diffusion approach leads to significant training instability, suboptimal performance, and severe length bias issues. To address these challenges, we design a novel training paradigm that combines an initial autoregressive phase with a subsequent diffusion phase. This approach yields the Dimple-7B model, trained on the same dataset and using a similar training pipeline as LLaVA-NEXT. Dimple-7B ultimately surpasses LLaVA-NEXT in performance by 3.9%, demonstrating that DMLLM can achieve performance comparable to that of autoregressive models. To improve inference efficiency, we propose a decoding strategy termed confident decoding, which dynamically adjusts the number of tokens generated at each step, significantly reducing the number of generation iterations. In autoregressive models, the number of forward iterations during generation equals the response length. With confident decoding, however, the number of iterations needed by Dimple is even only text{response length}{3}. We also re-implement the prefilling technique in autoregressive models and demonstrate that it does not significantly impact performance on most benchmark evaluations, while offering a speedup of 1.5x to 7x. Additionally, we explore Dimple's capability to precisely control its response using structure priors. These priors enable structured responses in a manner distinct from instruction-based or chain-of-thought prompting, and allow fine-grained control over response format and length, which is difficult to achieve in autoregressive models. Overall, this work validates the feasibility and advantages of DMLLM and enhances its inference efficiency and controllability. Code and models are available at https://github.com/yu-rp/Dimple.