https://huggingface.co/papers/2406.11794 from the amazing \n\n@vaishaal\n\t and many many co-authors (dataset is also here: https://huggingface.co/datasets/mlfoundations/dclm-baseline-1.0)\nhttps://huggingface.co/papers/2406.08446 from the AllenAI team \n\n@SaveBertAndGpt\n\t \n\n@OyvindTafjord\n\t \n\n@baileyk\n\t \n\n@JesseDodge\n\t \n\n","updatedAt":"2024-06-26T12:25:48.567Z","author":{"_id":"5df7e9e5da6d0311fd3d53f9","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1583857746553-5df7e9e5da6d0311fd3d53f9.jpeg","fullname":"Thomas Wolf","name":"thomwolf","type":"user","isPro":true,"isHf":true,"isHfAdmin":false,"isMod":false,"followerCount":1487}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.8047207593917847},"editors":["thomwolf"],"editorAvatarUrls":["https://cdn-avatars.huggingface.co/v1/production/uploads/1583857746553-5df7e9e5da6d0311fd3d53f9.jpeg"],"reactions":[{"reaction":"👍","users":["AdinaY","alielfilali01","mertege","julien-c"],"count":4},{"reaction":"🤗","users":["AdinaY","alielfilali01"],"count":2}],"isReport":false}},{"id":"667c14c9986bec0ba40c5417","author":{"_id":"5e9ecfc04957053f60648a3e","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1594214747713-5e9ecfc04957053f60648a3e.png","fullname":"Quentin Lhoest","name":"lhoestq","type":"user","isPro":true,"isHf":true,"isHfAdmin":false,"isMod":false,"followerCount":371},"createdAt":"2024-06-26T13:16:57.000Z","type":"comment","data":{"edited":false,"hidden":false,"latest":{"raw":"Some Spaces demo to query the dataset in SQL:\n- https://huggingface.co/spaces/lhoestq/fineweb-sql\n- https://huggingface.co/spaces/lhoestq/fineweb-edu-sql","html":"
Some Spaces demo to query the dataset in SQL:
\n- \n
- https://huggingface.co/spaces/lhoestq/fineweb-sql \n
- https://huggingface.co/spaces/lhoestq/fineweb-edu-sql \n
\n\n@librarian-bot\n\t recommend
\n","updatedAt":"2024-11-17T19:25:34.298Z","author":{"_id":"60107b385ac3e86b3ea4fc34","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1627505688463-60107b385ac3e86b3ea4fc34.jpeg","fullname":"Daniel van Strien","name":"davanstrien","type":"user","isPro":true,"isHf":true,"isHfAdmin":false,"isMod":false,"followerCount":707}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.7918877601623535},"editors":["davanstrien"],"editorAvatarUrls":["https://cdn-avatars.huggingface.co/v1/production/uploads/1627505688463-60107b385ac3e86b3ea4fc34.jpeg"],"reactions":[],"isReport":false},"replies":[{"id":"673a433119cbbe30913649d4","author":{"_id":"63d3e0e8ff1384ce6c5dd17d","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1674830754237-63d3e0e8ff1384ce6c5dd17d.jpeg","fullname":"Librarian Bot (Bot)","name":"librarian-bot","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":264},"createdAt":"2024-11-17T19:25:37.000Z","type":"comment","data":{"edited":false,"hidden":false,"latest":{"raw":"This is an automated message from the [Librarian Bot](https://huggingface.co/librarian-bots). I found the following papers similar to this paper. \n\nThe following papers were recommended by the Semantic Scholar API \n\n* [SWEb: A Large Web Dataset for the Scandinavian Languages](https://huggingface.co/papers/2410.04456) (2024)\n* [InfiMM-WebMath-40B: Advancing Multimodal Pre-Training for Enhanced Mathematical Reasoning](https://huggingface.co/papers/2409.12568) (2024)\n* [CCI3.0-HQ: a large-scale Chinese dataset of high quality designed for pre-training large language models](https://huggingface.co/papers/2410.18505) (2024)\n* [Multilingual Pretraining Using a Large Corpus Machine-Translated from a Single Source Language](https://huggingface.co/papers/2410.23956) (2024)\n* [Data Processing for the OpenGPT-X Model Family](https://huggingface.co/papers/2410.08800) (2024)\n\n\n Please give a thumbs up to this comment if you found it helpful!\n\n If you want recommendations for any Paper on Hugging Face checkout [this](https://huggingface.co/spaces/librarian-bots/recommend_similar_papers) Space\n\n You can directly ask Librarian Bot for paper recommendations by tagging it in a comment: `@librarian-bot recommend`","html":"This is an automated message from the Librarian Bot. I found the following papers similar to this paper.
\nThe following papers were recommended by the Semantic Scholar API
\n- \n
- SWEb: A Large Web Dataset for the Scandinavian Languages (2024) \n
- InfiMM-WebMath-40B: Advancing Multimodal Pre-Training for Enhanced Mathematical Reasoning (2024) \n
- CCI3.0-HQ: a large-scale Chinese dataset of high quality designed for pre-training large language models (2024) \n
- Multilingual Pretraining Using a Large Corpus Machine-Translated from a Single Source Language (2024) \n
- Data Processing for the OpenGPT-X Model Family (2024) \n
Please give a thumbs up to this comment if you found it helpful!
\nIf you want recommendations for any Paper on Hugging Face checkout this Space
\n You can directly ask Librarian Bot for paper recommendations by tagging it in a comment: \n\n@librarian-bot\n\t recommend
Papers
arxiv:2406.17557
The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale
Authors:
Abstract
FineWeb, a 15-trillion token dataset from Common Crawl snapshots, outperforms other open datasets in LLM training, and its educational subset, FineWeb-Edu, significantly improves performance on knowledge and reasoning benchmarks.
AI-generated summary
The performance of a large language model (LLM) depends heavily on the quality and size of its pretraining dataset. However, the pretraining datasets for state-of-the-art open LLMs like Llama 3 and Mixtral are not publicly available and very little is known about how they were created. In this work, we introduce FineWeb, a 15-trillion token dataset derived from 96 Common Crawl snapshots that produces better-performing LLMs than other open pretraining datasets. To advance the understanding of how best to curate high-quality pretraining datasets, we carefully document and ablate all of the design choices used in FineWeb, including in-depth investigations of deduplication and filtering strategies. In addition, we introduce FineWeb-Edu, a 1.3-trillion token collection of educational text filtered from FineWeb. LLMs pretrained on FineWeb-Edu exhibit dramatically better performance on knowledge- and reasoning-intensive benchmarks like MMLU and ARC. Along with our datasets, we publicly release our data curation codebase and all of the models trained during our ablation experiments.
Community
Paper to the latest and most popular pretraining dataset.
Cool paper!
Related recent work to continue the readings on top of this work:
- https://huggingface.co/papers/2406.11794 from the amazing @vaishaal and many many co-authors (dataset is also here: https://huggingface.co/datasets/mlfoundations/dclm-baseline-1.0)
- https://huggingface.co/papers/2406.08446 from the AllenAI team @SaveBertAndGpt @OyvindTafjord @baileyk @JesseDodge
Some Spaces demo to query the dataset in SQL:
This is an automated message from the Librarian Bot. I found the following papers similar to this paper.
The following papers were recommended by the Semantic Scholar API
- SWEb: A Large Web Dataset for the Scandinavian Languages (2024)
- InfiMM-WebMath-40B: Advancing Multimodal Pre-Training for Enhanced Mathematical Reasoning (2024)
- CCI3.0-HQ: a large-scale Chinese dataset of high quality designed for pre-training large language models (2024)
- Multilingual Pretraining Using a Large Corpus Machine-Translated from a Single Source Language (2024)
- Data Processing for the OpenGPT-X Model Family (2024)
Please give a thumbs up to this comment if you found it helpful!
If you want recommendations for any Paper on Hugging Face checkout this Space
You can directly ask Librarian Bot for paper recommendations by tagging it in a comment:
@librarian-bot
recommend