@librarian-bot\n\t recommend

\n","updatedAt":"2024-06-14T20:48:28.917Z","author":{"_id":"646b8e6f31968a60a0201a12","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/646b8e6f31968a60a0201a12/SU2Gs1NPuk1zoXHwFHl0U.jpeg","fullname":")))?!?(((","name":"stereoplegic","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":3762}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.7918877601623535},"editors":["stereoplegic"],"editorAvatarUrls":["https://cdn-avatars.huggingface.co/v1/production/uploads/646b8e6f31968a60a0201a12/SU2Gs1NPuk1zoXHwFHl0U.jpeg"],"reactions":[],"isReport":false},"replies":[{"id":"666caca389e21df7d4c9310b","author":{"_id":"63d3e0e8ff1384ce6c5dd17d","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1674830754237-63d3e0e8ff1384ce6c5dd17d.jpeg","fullname":"Librarian Bot (Bot)","name":"librarian-bot","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":264},"createdAt":"2024-06-14T20:48:35.000Z","type":"comment","data":{"edited":false,"hidden":false,"latest":{"raw":"This is an automated message from the [Librarian Bot](https://huggingface.co/librarian-bots). I found the following papers similar to this paper. \n\nThe following papers were recommended by the Semantic Scholar API \n\n* [LoCoCo: Dropping In Convolutions for Long Context Compression](https://huggingface.co/papers/2406.05317) (2024)\n* [SelfCP: Compressing Long Prompt to 1/12 Using the Frozen Large Language Model Itself](https://huggingface.co/papers/2405.17052) (2024)\n* [Retaining Key Information under High Compression Ratios: Query-Guided Compressor for LLMs](https://huggingface.co/papers/2406.02376) (2024)\n* [HMT: Hierarchical Memory Transformer for Long Context Language Processing](https://huggingface.co/papers/2405.06067) (2024)\n* [LongSkywork: A Training Recipe for Efficiently Extending Context Length in Large Language Models](https://huggingface.co/papers/2406.00605) (2024)\n\n\n Please give a thumbs up to this comment if you found it helpful!\n\n If you want recommendations for any Paper on Hugging Face checkout [this](https://huggingface.co/spaces/librarian-bots/recommend_similar_papers) Space\n\n You can directly ask Librarian Bot for paper recommendations by tagging it in a comment: `@librarian-bot recommend`","html":"

This is an automated message from the Librarian Bot. I found the following papers similar to this paper.

The following papers were recommended by the Semantic Scholar API

Please give a thumbs up to this comment if you found it helpful!

If you want recommendations for any Paper on Hugging Face checkout this Space

You can directly ask Librarian Bot for paper recommendations by tagging it in a comment: \n\n@librarian-bot\n\t recommend

\n","updatedAt":"2024-06-14T20:48:35.728Z","author":{"_id":"63d3e0e8ff1384ce6c5dd17d","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1674830754237-63d3e0e8ff1384ce6c5dd17d.jpeg","fullname":"Librarian Bot (Bot)","name":"librarian-bot","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":264}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.7483028173446655},"editors":["librarian-bot"],"editorAvatarUrls":["https://cdn-avatars.huggingface.co/v1/production/uploads/1674830754237-63d3e0e8ff1384ce6c5dd17d.jpeg"],"reactions":[],"isReport":false,"parentCommentId":"666cac9ca119281ee0ff42af"}}]}],"primaryEmailConfirmed":false,"paper":{"id":"2406.06110","authors":[{"_id":"6667b3350e69bd1af6847673","name":"Chensen Huang","hidden":false},{"_id":"6667b3350e69bd1af6847674","name":"Guibo Zhu","hidden":false},{"_id":"6667b3350e69bd1af6847675","name":"Xuepeng Wang","hidden":false},{"_id":"6667b3350e69bd1af6847676","name":"Yifei Luo","hidden":false},{"_id":"6667b3350e69bd1af6847677","name":"Guojing Ge","hidden":false},{"_id":"6667b3350e69bd1af6847678","name":"Haoran Chen","hidden":false},{"_id":"6667b3350e69bd1af6847679","name":"Dong Yi","hidden":false},{"_id":"6667b3350e69bd1af684767a","name":"Jinqiao Wang","hidden":false}],"publishedAt":"2024-06-10T08:50:59.000Z","title":"Recurrent Context Compression: Efficiently Expanding the Context Window\n of LLM","summary":"To extend the context length of Transformer-based large language models\n(LLMs) and improve comprehension capabilities, we often face limitations due to\ncomputational resources and bounded memory storage capacity. This work\nintroduces a method called Recurrent Context Compression (RCC), designed to\nefficiently expand the context window length of LLMs within constrained storage\nspace. We also investigate the issue of poor model responses when both\ninstructions and context are compressed in downstream tasks, and propose an\ninstruction reconstruction method to mitigate this problem. We validated the\neffectiveness of our approach on multiple tasks, achieving a compression rate\nof up to 32x on text reconstruction tasks with a BLEU4 score close to 0.95, and\nnearly 100\\% accuracy on a passkey retrieval task with a sequence length of 1M.\nFinally, our method demonstrated competitive performance in long-text\nquestion-answering tasks compared to non-compressed methods, while\nsignificantly saving storage resources in long-text inference tasks. Our code,\nmodels, and demo are available at https://github.com/WUHU-G/RCC_Transformer","upvotes":1,"discussionId":"6667b3360e69bd1af68476d8","ai_summary":"A Recurrent Context Compression (RCC) method extending the context length of Transformer-based models achieves high compression rates and maintains performance in various tasks, including question-answering and passkey retrieval.","ai_keywords":["Transformer-based models","Recurrent Context Compression (RCC)","context window","memory storage capacity","instruction reconstruction","BLEU4 score","long-text question-answering","long-text inference"]},"canReadDatabase":false,"canManagePapers":false,"canSubmit":false,"hasHfLevelAccess":false,"upvoted":false,"upvoters":[{"_id":"65decc75beffeb39ba679eba","avatarUrl":"/avatars/735b678bd5863a0c1b1bdd3bbf8858fa.svg","isPro":true,"fullname":"r","user":"oceansweep","type":"user"}],"acceptLanguages":["*"]}">

Papers

arxiv:2406.06110

Recurrent Context Compression: Efficiently Expanding the Context Window of LLM

Published on Jun 10, 2024

Upvote

Authors:

Abstract

A Recurrent Context Compression (RCC) method extending the context length of Transformer-based models achieves high compression rates and maintains performance in various tasks, including question-answering and passkey retrieval.

AI-generated summary

To extend the context length of Transformer-based large language models (LLMs) and improve comprehension capabilities, we often face limitations due to computational resources and bounded memory storage capacity. This work introduces a method called Recurrent Context Compression (RCC), designed to efficiently expand the context window length of LLMs within constrained storage space. We also investigate the issue of poor model responses when both instructions and context are compressed in downstream tasks, and propose an instruction reconstruction method to mitigate this problem. We validated the effectiveness of our approach on multiple tasks, achieving a compression rate of up to 32x on text reconstruction tasks with a BLEU4 score close to 0.95, and nearly 100\% accuracy on a passkey retrieval task with a sequence length of 1M. Finally, our method demonstrated competitive performance in long-text question-answering tasks compared to non-compressed methods, while significantly saving storage resources in long-text inference tasks. Our code, models, and demo are available at https://github.com/WUHU-G/RCC_Transformer

View arXiv page View PDF Add to collection