https://huggingface.co/TechxGenus/Mini-Jamba\n","updatedAt":"2024-04-01T16:54:28.469Z","author":{"_id":"63ff78e2b09f82a81a1d6f52","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/63ff78e2b09f82a81a1d6f52/6SnqEfGDRH25sqJXAV4mf.png","fullname":"Sciumo","name":"Sciumo","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.8425146341323853},"editors":["Sciumo"],"editorAvatarUrls":["https://cdn-avatars.huggingface.co/v1/production/uploads/63ff78e2b09f82a81a1d6f52/6SnqEfGDRH25sqJXAV4mf.png"],"reactions":[{"reaction":"🔥","users":["clem","emanuelevivoli"],"count":2}],"isReport":false}},{"id":"660b5e645814b3dfc3e6c94c","author":{"_id":"63d3e0e8ff1384ce6c5dd17d","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1674830754237-63d3e0e8ff1384ce6c5dd17d.jpeg","fullname":"Librarian Bot (Bot)","name":"librarian-bot","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":264},"createdAt":"2024-04-02T01:24:52.000Z","type":"comment","data":{"edited":false,"hidden":false,"latest":{"raw":"This is an automated message from the [Librarian Bot](https://huggingface.co/librarian-bots). I found the following papers similar to this paper. \n\nThe following papers were recommended by the Semantic Scholar API \n\n* [DenseMamba: State Space Models with Dense Hidden Connection for Efficient Large Language Models](https://huggingface.co/papers/2403.00818) (2024)\n* [Cobra: Extending Mamba to Multi-Modal Large Language Model for Efficient Inference](https://huggingface.co/papers/2403.14520) (2024)\n* [Dynamic Memory Compression: Retrofitting LLMs for Accelerated Inference](https://huggingface.co/papers/2403.09636) (2024)\n* [ALISA: Accelerating Large Language Model Inference via Sparsity-Aware KV Caching](https://huggingface.co/papers/2403.17312) (2024)\n* [Can Mamba Learn How to Learn? A Comparative Study on In-Context Learning Tasks](https://huggingface.co/papers/2402.04248) (2024)\n\n\n Please give a thumbs up to this comment if you found it helpful!\n\n If you want recommendations for any Paper on Hugging Face checkout [this](https://huggingface.co/spaces/librarian-bots/recommend_similar_papers) Space\n\n You can directly ask Librarian Bot for paper recommendations by tagging it in a comment: `@librarian-bot recommend`","html":"
This is an automated message from the Librarian Bot. I found the following papers similar to this paper.
\nThe following papers were recommended by the Semantic Scholar API
\n- \n
- DenseMamba: State Space Models with Dense Hidden Connection for Efficient Large Language Models (2024) \n
- Cobra: Extending Mamba to Multi-Modal Large Language Model for Efficient Inference (2024) \n
- Dynamic Memory Compression: Retrofitting LLMs for Accelerated Inference (2024) \n
- ALISA: Accelerating Large Language Model Inference via Sparsity-Aware KV Caching (2024) \n
- Can Mamba Learn How to Learn? A Comparative Study on In-Context Learning Tasks (2024) \n
Please give a thumbs up to this comment if you found it helpful!
\nIf you want recommendations for any Paper on Hugging Face checkout this Space
\n You can directly ask Librarian Bot for paper recommendations by tagging it in a comment: \n\n@librarian-bot\n\t recommend
I have featured this paper on my blog. You can read it at https://rb.gy/6r0rs3
\n","updatedAt":"2024-04-10T10:46:51.078Z","author":{"_id":"648a210e9da3cc3506961585","avatarUrl":"/avatars/808e9d7ac99837fe79169d0b8d49c366.svg","fullname":"Ajith V Prabhakar","name":"ajithprabhakar","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":2}},"numEdits":1,"identifiedLanguage":{"language":"en","probability":0.9560481309890747},"editors":["ajithprabhakar"],"editorAvatarUrls":["/avatars/808e9d7ac99837fe79169d0b8d49c366.svg"],"reactions":[],"isReport":false}},{"id":"6664c5298a270cedd5252325","author":{"_id":"6186ddf6a7717cb375090c01","avatarUrl":"/avatars/716b6a7d1094c8036b2a8a7b9063e8aa.svg","fullname":"Julien BLANCHON","name":"blanchon","type":"user","isPro":true,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":142},"createdAt":"2024-06-08T20:55:05.000Z","type":"comment","data":{"edited":false,"hidden":false,"latest":{"raw":"# Jamba: Revolutionizing Language Models with a Hybrid Transformer Approach\n\nhttps://cdn-uploads.huggingface.co/production/uploads/6186ddf6a7717cb375090c01/7xsek0yVIC60sJsx1ce3M.mp4 \n\n## Links 🔗:\n👉 Subscribe: https://www.youtube.com/@Arxflix\n👉 Twitter: https://x.com/arxflix\n👉 LMNT (Partner): https://lmnt.com/\n\n\nBy Arxflix\n","html":"Jamba: Revolutionizing Language Models with a Hybrid Transformer Approach
\n \n\nLinks 🔗:
\n👉 Subscribe: https://www.youtube.com/@Arxflix
👉 Twitter: https://x.com/arxflix
👉 LMNT (Partner): https://lmnt.com/
By Arxflix
Abstract
Jamba, a hybrid Transformer-Mamba MoE model, achieves state-of-the-art performance with high throughput and minimal memory usage, supporting large context lengths.
AI-generated summary
We present Jamba, a new base large language model based on a novel hybrid Transformer-Mamba mixture-of-experts (MoE) architecture. Specifically, Jamba interleaves blocks of Transformer and Mamba layers, enjoying the benefits of both model families. MoE is added in some of these layers to increase model capacity while keeping active parameter usage manageable. This flexible architecture allows resource- and objective-specific configurations. In the particular configuration we have implemented, we end up with a powerful model that fits in a single 80GB GPU. Built at large scale, Jamba provides high throughput and small memory footprint compared to vanilla Transformers, and at the same time state-of-the-art performance on standard language model benchmarks and long-context evaluations. Remarkably, the model presents strong results for up to 256K tokens context length. We study various architectural decisions, such as how to combine Transformer and Mamba layers, and how to mix experts, and show that some of them are crucial in large scale modeling. We also describe several interesting properties of these architectures which the training and evaluation of Jamba have revealed, and plan to release checkpoints from various ablation runs, to encourage further exploration of this novel architecture. We make the weights of our implementation of Jamba publicly available under a permissive license.
Community
good
This is an automated message from the Librarian Bot. I found the following papers similar to this paper.
The following papers were recommended by the Semantic Scholar API
- DenseMamba: State Space Models with Dense Hidden Connection for Efficient Large Language Models (2024)
- Cobra: Extending Mamba to Multi-Modal Large Language Model for Efficient Inference (2024)
- Dynamic Memory Compression: Retrofitting LLMs for Accelerated Inference (2024)
- ALISA: Accelerating Large Language Model Inference via Sparsity-Aware KV Caching (2024)
- Can Mamba Learn How to Learn? A Comparative Study on In-Context Learning Tasks (2024)
Please give a thumbs up to this comment if you found it helpful!
If you want recommendations for any Paper on Hugging Face checkout this Space
You can directly ask Librarian Bot for paper recommendations by tagging it in a comment:
@librarian-bot
recommend
Jamba: Revolutionizing Language Models with a Hybrid Transformer Approach
Links 🔗:
👉 Subscribe: https://www.youtube.com/@Arxflix
👉 Twitter: https://x.com/arxflix
👉 LMNT (Partner): https://lmnt.com/
By Arxflix
Models citing this paper 4
Datasets citing this paper 0
No dataset linking this paper
Cite arxiv.org/abs/2403.19887 in a dataset README.md to link it from this page.