From Chunks to Blocks: Accelerating Uploads and Downloads on the Hub

Published February 12, 2025

Update on GitHub

Upvote

xet-team

xet-team

xet-team

Content-defined chunking (CDC) plays a central role in enabling deduplication within a Xet-backed repository. The idea is straightforward: break each file’s data into chunks, store only unique ones, reap the benefits.

In practice, it's more complex. If we focused solely on maximizing deduplication, the design would call for the smallest possible chunk size. By doing that, we’d create significant overheads for the infrastructure and the builders on the Hub.

On Hugging Face's Xet team, we're bringing CDC from theory to production to deliver faster uploads and downloads to AI builders (by a factor of 2-3x in some cases). Our guiding principle is simple: enable rapid experimentation and collaboration for teams building and iterating on models and datasets. This means focusing on more than just deduplication; we’re optimizing how data moves across the network, how it’s stored, and the entire development experience.

The Realities of Scaling Deduplication

Imagine uploading a 200GB repository to the Hub. Today, there are a number of ways to do this, but all use a file-centric approach. To bring faster file transfers to the Hub, we've open-sourced xet-core and hf_xet, an integration with huggingface_hub which uses a chunk-based approach written in Rust.

If you consider a 200GB repository with unique chunks, that's 3 million entries (at ~64KB per chunk) in the content-addressed store (CAS) backing all repositories. If a new version of a model is uploaded or a branch in the repository is created with different data, more unique chunks are added, driving up the entries in the CAS.

With nearly 45PB across 2 million model, dataset, and space repositories on the Hub, a purely chunk-based approach could incur 690 billion chunks. Managing this volume of content using only chunks is simply not viable due to:

Network Overheads: If each chunk is downloaded or uploaded individually, millions of requests are generated on each upload and download, overwhelming both client and server. Even batching queries simply shifts the problem to the storage layer.
Infrastructure Overheads: A naive CAS that tracks chunks individually would require billions of entries, leading to steep monthly bills on services like DynamoDB or S3. At Hugging Face’s scale, this quickly adds up.

In short, network requests balloon, databases struggle to manage the metadata, and the cost of orchestrating each chunk skyrockets all while you wait for your files to transfer.

Design Principles for Deduplication at Scale

These challenges lead to a key realization:

Deduplication is a performance optimization, not the final goal.

The final goal is to improve the experience of builders iterating and collaborating on models and datasets. The system components from the client to the storage layer do not need to guarantee deduplication. Instead, they leverage deduplication as one tool among many to aid in this.

By loosening the deduplication constraint, we naturally arrive at a second design principle:

Avoid communication or storage strategies that scale 1:1 with the number of chunks.

What does this mean? We scale with aggregation.

Scaling Deduplication with Aggregation

Aggregation takes chunks and groups them, referencing them intelligently in ways that provide clever (and practical) benefits:

Blocks: Instead of transferring and storing chunks, we bundle data together in blocks of up to 64MB after deduplication. Blocks are still content-addressed, but this reduces CAS entries by a factor of 1,000.
Shards: Shards provide the mapping between files and chunks (referencing blocks as they do so). This allows us to identify which parts of a file have changed, referencing shards generated from past uploads. When chunks are already known to exist in the CAS, they’re skipped, slashing unnecessary transfers and queries.

Together, blocks and shards unlock significant benefits. However, when someone uploads a new file, how do we know if a chunk has been uploaded before so we can eliminate an unnecessary request? Performing a network query for every chunk is not scalable and goes against the “no 1:1” principle we mentioned above.

The solution is key chunks which are a 0.1% subset of all chunks selected with a simple modulo condition based on the chunk hash. We provide a global index over these key chunks and the shards they are found in, so that when the chunk is queried, the related shard is returned to provide local deduplication. This allows us to leverage the principles of spatial locality. If a key chunk is referenced in a shard, it’s likely that other similar chunk references are available in the same shard. This further improves deduplication and reduces network and database requests.

Key chunks

Aggregated Deduplication in Practice

The Hub currently stores over 3.5PB of .gguf files, most of which are quantized versions of other models on the Hub. Quantized models represent an interesting opportunity for deduplication due to the nature of quantization where values are restricted to a smaller integer range and scaled. This restricts the range of values in the weight matrices, naturally leading to more repetition. Additionally, many repositories of quantized models store multiple different variants (e.g., Q4_K, Q3_K, Q5_K) with a great deal of overlap.

A good example of this in practice is bartowski/gemma-2-9b-it-GGUF which contains 29 quantizations of google/gemma-2-9b-it totalling 191GB. To upload, we use hf_xet integrated with huggingface_hub to perform chunk-level deduplication locally then aggregate and store data at the block level.

Once uploaded, we can start to see some cool patterns! We’ve included a visualization that shows the deduplication ratio for each block. The darker the block, the more frequently parts of it are referenced across model versions. If you go to the Space hosting this visualization, hovering over any heatmap cell highlights all references to the block in orange across all models while clicking on a cell will select all other files that share blocks:

Quantization deduplication visualization

A single block of deduplication might only represent a few MB of savings, but as you can see there are many overlapping blocks! With this many blocks that quickly adds up. Instead of uploading 191GB, the Xet-backed version of the gemma-2-9b-it-GGUF repository stores 1515 unique blocks for a total of approximately 97GB to our test CAS environment (a savings of ~94GB).

While the storage improvements are significant, the real benefit is what this means for contributors to the Hub. At 50MB/s, the deduplication optimizations amount to a four hour difference in upload time; a speedup of nearly 2x:

Repo	Stored Size	Upload Time @ 50MB/s
Original	191 GB	509 minutes
Xet-backed	97 GB	258 minutes

Similarly, local chunk caching significantly speeds up downloads. If a file is changed or a new quantization is added that has significant overlap with the local chunk cache, you won’t have to re-download any chunks that are unchanged. This contrasts to the file-based approach where the entirety of the new or updated file must be downloaded.

Taken together, this demonstrates how local chunk-level deduplication paired with block-level aggregation dramatically streamlines not just storage, but developing on the Hub. By providing this level of efficiency in file transfers, AI builders can move faster, iterate quickly, and worry less about hitting infrastructure bottlenecks. For anyone pushing large files to the Hub (whether you're pushing a new model quantization or an updated version of a training set) this helps you shift focus to building and sharing, rather than waiting and troubleshooting.

We’re fast at work, rolling out the first Xet-backed repositories in the coming weeks and months! As we do that, we will be releasing more updates to bring these speeds to every builder on the Hub to make file transfers feel invisible.

From Files to Chunks: Improving Hugging Face Storage Efficiency

By November 20, 2024 • 63

Parquet Content-Defined Chunking

By July 25, 2025 • 64

Community

ngxson/kokoro-podcast-generator, using DeepSeek-R1 and Kokoro-TTS

\n","updatedAt":"2025-02-17T22:13:54.430Z","author":{"_id":"63ca214abedad7e2bf1d1517","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1674191139776-noauth.png","fullname":"Xuan-Son Nguyen","name":"ngxson","type":"user","isPro":false,"isHf":true,"isHfAdmin":false,"isMod":false,"followerCount":363}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.745898425579071},"editors":["ngxson"],"editorAvatarUrls":["https://cdn-avatars.huggingface.co/v1/production/uploads/1674191139776-noauth.png"],"reactions":[{"reaction":"🔥","users":["rajatarya","Bradarr"],"count":2},{"reaction":"🤗","users":["rajatarya"],"count":1}],"isReport":false}},{"id":"67b435c8bffd44cc85a0ade1","author":{"_id":"66a07001440c06d64352cfc8","avatarUrl":"/avatars/93a7d12ac4d300a56bfb4bdfc54e6d33.svg","fullname":"liming","name":"largegpt","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false},"createdAt":"2025-02-18T07:24:56.000Z","type":"comment","data":{"edited":true,"hidden":false,"latest":{"raw":"How can we find the chunk content using chunk hash? The CAS system only stores \"block_hash -> block_content\", Where does the map of chunk to block?\n\nwhat does the shards store? Is it \"file_name, shard_id, chunk_hash, block_hash\"?","html":"

How can we find the chunk content using chunk hash? The CAS system only stores \"block_hash -> block_content\", Where does the map of chunk to block?

what does the shards store? Is it \"file_name, shard_id, chunk_hash, block_hash\"?

\n","updatedAt":"2025-02-18T07:25:14.971Z","author":{"_id":"66a07001440c06d64352cfc8","avatarUrl":"/avatars/93a7d12ac4d300a56bfb4bdfc54e6d33.svg","fullname":"liming","name":"largegpt","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false}},"numEdits":1,"identifiedLanguage":{"language":"en","probability":0.8209852576255798},"editors":["largegpt"],"editorAvatarUrls":["/avatars/93a7d12ac4d300a56bfb4bdfc54e6d33.svg"],"reactions":[],"isReport":false},"replies":[{"id":"67b5037e864242a26c23e881","author":{"_id":"64beb7a2c733e8552ffd63b3","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/64beb7a2c733e8552ffd63b3/ZI_DOExd737quSFZLmx58.jpeg","fullname":"Sam Horradarn","name":"sirahd","type":"user","isPro":false,"isHf":true,"isHfAdmin":false,"isMod":false,"followerCount":37},"createdAt":"2025-02-18T22:02:38.000Z","type":"comment","data":{"edited":false,"hidden":false,"latest":{"raw":"> How can we find the chunk content using chunk hash?\n\nChunk hash is calculated via content-defined chunking (CDC), which means that if two chunks have the same content they will share the same hash. CDC removes the need to store the mapping between chunk hash -> chunk content because we know if two chunks share the same hash, they will have identical content.\n\n> The CAS system only stores \"block_hash -> block_content\", Where does the map of chunk to block?\n\nThis is explained in the \"key chunks\" section in the blog post above. Essentially we only store a tiny subset of chunk -> block by leveraging spatial locality in the file. Trying to store every mapping of chunk -> block can get impractical very quickly.\n\n> what does the shards store? Is it \"file_name, shard_id, chunk_hash, block_hash\"\n\nYou can think of the shards as storing mappings between file (identified via file hash) to list of chunks that make up the file.\n\nI hope this help explains our underlying tech better!","html":"

\n
How can we find the chunk content using chunk hash?
\n

Chunk hash is calculated via content-defined chunking (CDC), which means that if two chunks have the same content they will share the same hash. CDC removes the need to store the mapping between chunk hash -> chunk content because we know if two chunks share the same hash, they will have identical content.

\n
The CAS system only stores \"block_hash -> block_content\", Where does the map of chunk to block?
\n

This is explained in the \"key chunks\" section in the blog post above. Essentially we only store a tiny subset of chunk -> block by leveraging spatial locality in the file. Trying to store every mapping of chunk -> block can get impractical very quickly.

\n
what does the shards store? Is it \"file_name, shard_id, chunk_hash, block_hash\"
\n

You can think of the shards as storing mappings between file (identified via file hash) to list of chunks that make up the file.

I hope this help explains our underlying tech better!

\n","updatedAt":"2025-02-18T22:02:38.999Z","author":{"_id":"64beb7a2c733e8552ffd63b3","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/64beb7a2c733e8552ffd63b3/ZI_DOExd737quSFZLmx58.jpeg","fullname":"Sam Horradarn","name":"sirahd","type":"user","isPro":false,"isHf":true,"isHfAdmin":false,"isMod":false,"followerCount":37}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.924042284488678},"editors":["sirahd"],"editorAvatarUrls":["https://cdn-avatars.huggingface.co/v1/production/uploads/64beb7a2c733e8552ffd63b3/ZI_DOExd737quSFZLmx58.jpeg"],"reactions":[{"reaction":"❤️","users":["jsulz"],"count":1}],"isReport":false,"parentCommentId":"67b435c8bffd44cc85a0ade1"}},{"id":"67b54eae92ed469d5f99f6ef","author":{"_id":"66a07001440c06d64352cfc8","avatarUrl":"/avatars/93a7d12ac4d300a56bfb4bdfc54e6d33.svg","fullname":"liming","name":"largegpt","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false},"createdAt":"2025-02-19T03:23:26.000Z","type":"comment","data":{"edited":true,"hidden":true,"hiddenBy":"","latest":{"raw":"This comment has been hidden","html":"This comment has been hidden","updatedAt":"2025-02-24T04:56:31.841Z","author":{"_id":"66a07001440c06d64352cfc8","avatarUrl":"/avatars/93a7d12ac4d300a56bfb4bdfc54e6d33.svg","fullname":"liming","name":"largegpt","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false}},"numEdits":0,"editors":[],"editorAvatarUrls":[],"reactions":[],"parentCommentId":"67b435c8bffd44cc85a0ade1"}}]},{"id":"67bbfc2194fcec47ae0e36bc","author":{"_id":"66a07001440c06d64352cfc8","avatarUrl":"/avatars/93a7d12ac4d300a56bfb4bdfc54e6d33.svg","fullname":"liming","name":"largegpt","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false},"createdAt":"2025-02-24T04:57:05.000Z","type":"comment","data":{"edited":false,"hidden":false,"latest":{"raw":"If we don't store every chunk hash to block hash, how can we download a model file. For example if a model does not have a key chunk, the shard info only contains \"file hash -> list of chunks\", the CAS only has \"block hash -> block content\", how can we know which block is needed ?\n","html":"

If we don't store every chunk hash to block hash, how can we download a model file. For example if a model does not have a key chunk, the shard info only contains \"file hash -> list of chunks\", the CAS only has \"block hash -> block content\", how can we know which block is needed ?

\n","updatedAt":"2025-02-24T04:57:05.089Z","author":{"_id":"66a07001440c06d64352cfc8","avatarUrl":"/avatars/93a7d12ac4d300a56bfb4bdfc54e6d33.svg","fullname":"liming","name":"largegpt","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.9205638766288757},"editors":["largegpt"],"editorAvatarUrls":["/avatars/93a7d12ac4d300a56bfb4bdfc54e6d33.svg"],"reactions":[],"isReport":false},"replies":[{"id":"67be432790cc736e2f8e6c4e","author":{"_id":"65d50e9ef9cbfa798c590004","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/65d50e9ef9cbfa798c590004/FlVe8chafigMfrPpMeJRL.jpeg","fullname":"Jared Sulzdorf","name":"jsulz","type":"user","isPro":true,"isHf":true,"isHfAdmin":false,"isMod":false,"followerCount":243},"createdAt":"2025-02-25T22:24:39.000Z","type":"comment","data":{"edited":true,"hidden":false,"latest":{"raw":"Great questions :) \n\n> For example if a model does not have a key chunk\n\nTwo points of clarification might be in order concerning the above point: \n1. Not all files will contain a key chunk; this is purely an optimization.\n2. Key chunks are used for deduplication on the upload path, not for downloads. They allow us to see if a file uploaded to a repository for the first time has any content in the global store. This allows us to deduplicate over the entirety of the storage. \n\nAs for your other question: \n> If we don't store every chunk hash to block hash, how can we download a model file\n\nWhen downloading a model file, a request is made to our services with the file hash. This is mapped to a list of block subranges. Logically these are chunks, but by storing the offsets we save on the metadata storage and ultimately many offsets will share boundaries inside a block (allowing us to group them together in a response) providing benefits when sending the content back to the client.","html":"

Great questions :)

\n
For example if a model does not have a key chunk
\n

Two points of clarification might be in order concerning the above point:

Not all files will contain a key chunk; this is purely an optimization.
Key chunks are used for deduplication on the upload path, not for downloads. They allow us to see if a file uploaded to a repository for the first time has any content in the global store. This allows us to deduplicate over the entirety of the storage.

As for your other question:

\n
If we don't store every chunk hash to block hash, how can we download a model file
\n

When downloading a model file, a request is made to our services with the file hash. This is mapped to a list of block subranges. Logically these are chunks, but by storing the offsets we save on the metadata storage and ultimately many offsets will share boundaries inside a block (allowing us to group them together in a response) providing benefits when sending the content back to the client.

\n","updatedAt":"2025-02-26T04:56:49.883Z","author":{"_id":"65d50e9ef9cbfa798c590004","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/65d50e9ef9cbfa798c590004/FlVe8chafigMfrPpMeJRL.jpeg","fullname":"Jared Sulzdorf","name":"jsulz","type":"user","isPro":true,"isHf":true,"isHfAdmin":false,"isMod":false,"followerCount":243}},"numEdits":2,"identifiedLanguage":{"language":"en","probability":0.9278646111488342},"editors":["jsulz"],"editorAvatarUrls":["https://cdn-avatars.huggingface.co/v1/production/uploads/65d50e9ef9cbfa798c590004/FlVe8chafigMfrPpMeJRL.jpeg"],"reactions":[],"isReport":false,"parentCommentId":"67bbfc2194fcec47ae0e36bc"}},{"id":"67d55ea8a074f6158e265304","author":{"_id":"61bf40824b4300d0fb0acf59","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1644224872623-61bf40824b4300d0fb0acf59.jpeg","fullname":"Leshem Choshen","name":"borgr","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":23},"createdAt":"2025-03-15T11:04:08.000Z","type":"comment","data":{"edited":false,"hidden":false,"latest":{"raw":"Is all this data saved in raw fp? Or compressed somehow?\n\n (yes I have an unhidden agenda of wondering due to zipNN lossless (open source) model compression and its relevance https://github.com/zipnn/zipnn)","html":"

Is all this data saved in raw fp? Or compressed somehow?

(yes I have an unhidden agenda of wondering due to zipNN lossless (open source) model compression and its relevance https://github.com/zipnn/zipnn)

\n","updatedAt":"2025-03-15T11:04:08.625Z","author":{"_id":"61bf40824b4300d0fb0acf59","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1644224872623-61bf40824b4300d0fb0acf59.jpeg","fullname":"Leshem Choshen","name":"borgr","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":23}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.8546241521835327},"editors":["borgr"],"editorAvatarUrls":["https://cdn-avatars.huggingface.co/v1/production/uploads/1644224872623-61bf40824b4300d0fb0acf59.jpeg"],"reactions":[],"isReport":false,"parentCommentId":"67bbfc2194fcec47ae0e36bc"}},{"id":"67d897cae3089f36355fb807","author":{"_id":"66ac094a8fc00b5c160d7da4","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/66ac094a8fc00b5c160d7da4/1-DnsQ0zlyTA-18bncHbt.jpeg","fullname":"yuchenglow","name":"yuchenglow","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":69},"createdAt":"2025-03-17T21:44:42.000Z","type":"comment","data":{"edited":false,"hidden":false,"latest":{"raw":"We LZ4 everything automatically, but when we do encounter a model, we also perform a format-agnostic byte grouping inspired by ZipNN before LZ4ing. This does empirically save about 20%.\n https://github.com/huggingface/xet-core/blob/main/cas_object/src/byte_grouping/bg4.rs","html":"

We LZ4 everything automatically, but when we do encounter a model, we also perform a format-agnostic byte grouping inspired by ZipNN before LZ4ing. This does empirically save about 20%.
https://github.com/huggingface/xet-core/blob/main/cas_object/src/byte_grouping/bg4.rs

\n","updatedAt":"2025-03-17T21:44:42.530Z","author":{"_id":"66ac094a8fc00b5c160d7da4","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/66ac094a8fc00b5c160d7da4/1-DnsQ0zlyTA-18bncHbt.jpeg","fullname":"yuchenglow","name":"yuchenglow","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":69}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.7416666746139526},"editors":["yuchenglow"],"editorAvatarUrls":["https://cdn-avatars.huggingface.co/v1/production/uploads/66ac094a8fc00b5c160d7da4/1-DnsQ0zlyTA-18bncHbt.jpeg"],"reactions":[],"isReport":false,"parentCommentId":"67bbfc2194fcec47ae0e36bc"}},{"id":"67d8990ea35ca2f0c093b59b","author":{"_id":"61bf40824b4300d0fb0acf59","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1644224872623-61bf40824b4300d0fb0acf59.jpeg","fullname":"Leshem Choshen","name":"borgr","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":23},"createdAt":"2025-03-17T21:50:06.000Z","type":"comment","data":{"edited":false,"hidden":false,"latest":{"raw":"That's great to know! Tx for sharing.","html":"

That's great to know! Tx for sharing.

\n","updatedAt":"2025-03-17T21:50:06.717Z","author":{"_id":"61bf40824b4300d0fb0acf59","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1644224872623-61bf40824b4300d0fb0acf59.jpeg","fullname":"Leshem Choshen","name":"borgr","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":23}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.9869318604469299},"editors":["borgr"],"editorAvatarUrls":["https://cdn-avatars.huggingface.co/v1/production/uploads/1644224872623-61bf40824b4300d0fb0acf59.jpeg"],"reactions":[],"isReport":false,"parentCommentId":"67bbfc2194fcec47ae0e36bc"}}]},{"id":"67d304341099f6e8e456536a","author":{"_id":"65d50e9ef9cbfa798c590004","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/65d50e9ef9cbfa798c590004/FlVe8chafigMfrPpMeJRL.jpeg","fullname":"Jared Sulzdorf","name":"jsulz","type":"user","isPro":true,"isHf":true,"isHfAdmin":false,"isMod":false,"followerCount":243},"createdAt":"2025-03-13T16:13:40.000Z","type":"comment","data":{"edited":false,"hidden":false,"latest":{"raw":"If you're reading this post and want to get your hands dirty, join the waitlist to get access! https://huggingface.co/join/xet","html":"

If you're reading this post and want to get your hands dirty, join the waitlist to get access! https://huggingface.co/join/xet

\n","updatedAt":"2025-03-13T16:13:40.787Z","author":{"_id":"65d50e9ef9cbfa798c590004","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/65d50e9ef9cbfa798c590004/FlVe8chafigMfrPpMeJRL.jpeg","fullname":"Jared Sulzdorf","name":"jsulz","type":"user","isPro":true,"isHf":true,"isHfAdmin":false,"isMod":false,"followerCount":243}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.9134101867675781},"editors":["jsulz"],"editorAvatarUrls":["https://cdn-avatars.huggingface.co/v1/production/uploads/65d50e9ef9cbfa798c590004/FlVe8chafigMfrPpMeJRL.jpeg"],"reactions":[],"isReport":false}}],"status":"open","isReport":false,"pinned":false,"locked":false,"collection":"canonical_blogs"},"contextAuthors":["jsulz","yuchenglow","znation","saba9"],"primaryEmailConfirmed":false,"discussionRole":0,"acceptLanguages":["*"],"withThread":true,"cardDisplay":false,"repoDiscussionsLocked":false}">

Taylor658

Feb 12

Nice to see that Xet backed repos are being rolled out on the Hub.

ngxson

Feb 17

📻 🎙️ Hey, I made a podcast about this blog post, check it out!

This podcast is generated via ngxson/kokoro-podcast-generator, using DeepSeek-R1 and Kokoro-TTS

largegpt

Feb 18

•

edited Feb 18

How can we find the chunk content using chunk hash? The CAS system only stores "block_hash -> block_content", Where does the map of chunk to block?

what does the shards store? Is it "file_name, shard_id, chunk_hash, block_hash"?

sirahd

Feb 18

How can we find the chunk content using chunk hash?

The CAS system only stores "block_hash -> block_content", Where does the map of chunk to block?

This is explained in the "key chunks" section in the blog post above. Essentially we only store a tiny subset of chunk -> block by leveraging spatial locality in the file. Trying to store every mapping of chunk -> block can get impractical very quickly.

what does the shards store? Is it "file_name, shard_id, chunk_hash, block_hash"

You can think of the shards as storing mappings between file (identified via file hash) to list of chunks that make up the file.

I hope this help explains our underlying tech better!

largegpt

Feb 24

If we don't store every chunk hash to block hash, how can we download a model file. For example if a model does not have a key chunk, the shard info only contains "file hash -> list of chunks", the CAS only has "block hash -> block content", how can we know which block is needed ?

jsulz

Article author Feb 25

•

edited Feb 26

Great questions :)

For example if a model does not have a key chunk

Two points of clarification might be in order concerning the above point:

Not all files will contain a key chunk; this is purely an optimization.
Key chunks are used for deduplication on the upload path, not for downloads. They allow us to see if a file uploaded to a repository for the first time has any content in the global store. This allows us to deduplicate over the entirety of the storage.

As for your other question:

If we don't store every chunk hash to block hash, how can we download a model file

jsulz

Article author Mar 13

If you're reading this post and want to get your hands dirty, join the waitlist to get access! https://huggingface.co/join/xet

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment

Upvote