In January of this year, Hugging Face's Xet Team deployed a new storage backend, and shortly after shifted ~6% of Hub downloads through the infrastructure. This represented a significant milestone, but it was just the beginning. In 6 months, 500,000 repositories holding 20 PB joined the move to Xet as the Hub outgrows Git LFS and transitions to a storage system that scales with the workloads of AI builders.
Today, more than 1 million people on the Hub are using Xet. In May, it became the default on the Hub for new users and organizations. With only a few dozen GitHub issues, forum threads, and Discord messages, this is perhaps the quietest migration of this magnitude.
How? For one, the team came prepared with years of experience building and supporting the content addressed store (CAS) and Rust client that provide the system's foundation. Without these pieces, Git LFS may still be the future on the Hub. However, the unsung heroes of this migration are:
Together, these components have allowed us to aggressively migrate PBs in the span of days without worrying about the impact to the Hub or the community. They're giving us the peace of mind to move even faster in the coming weeks and months (skip to the end 👇 to see what's coming).
In the early days of planning the migration to Xet, we made a few key design decisions:
Driven by our commitment to the community, these seemingly straightforward decisions had significant implications. Most importantly, we did not believe users and teams should have to immediately alter their workflow or download a new client to interact with Xet-enabled repositories.
If you have a Xet-aware client (e.g., hf-xet, the Xet integration with huggingface_hub), uploads and downloads pass through the entire Xet stack. The client either breaks up files into chunks using content defined chunking while uploading, or requests file reconstruction information when downloading. On upload, chunks are passed to CAS and stored in S3. During downloads, CAS provides the chunk ranges the client needs to request from S3 to reconstruct the file locally.
For older versions of huggingface_hub or huggingface.js, which do not support chunk-based file transfers, you can still download and upload to Xet repos, but these bytes take a different route. When a Xet-backed file is requested from the Hub along the resolve endpoint, the Git LFS Bridge constructs and returns a single presigned URL, mimicking the LFS protocol. The Bridge then does the work of reconstructing the file from the content held in S3 and returns it to the requester.
To see this in action, right click on the image above and open it in a new tab. The URL redirects from
https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/blog/migrating-the-hub-to-xet/bridge.png to one that begins with https://cas-bridge.xethub.hf.co/xet-bridge-us/.... You can also use curl -vL on the same URL to see the redirects in your terminal.
Meanwhile, when a non-Xet-aware client uploads a file, it is sent first to LFS storage then migrated to Xet. This “background migration process,” only briefly mentioned in our docs, powers both the migrations to Xet and upload backward compatibility. It is behind the migration of well over a dozen PBs of models and datasets and is keeping 500,000 repos in sync with Xet storage all without missing a beat.
Every time a file needs to be migrated from LFS to Xet, a webhook is triggered, pushing the event to a distributed queue where it is processed by an orchestrator. The orchestrator:
These migration workers then pick up the jobs and each pod:
In April, we tested this system's limits by reaching out to bartowski and asking if they wanted to test out Xet. With nearly 500 TB across 2,000 repos, bartowski's migration uncovered a few weak links:
/tmp and then moved into the shard cache. On our worker pods, however, /tmp and the Xet cache sat on different mount points. The move failed and the shard files were never removed. Eventually the disk filled, triggering a wave of No space left on device errors.Fixing this three-headed monster meant touching every layer - patching xet-core, resizing CAS, and beefing up the worker node specs. Fortunately, bartowski was game to work with us while every repo made its way to Xet. These same lessons powered the moves of the biggest storage users on the Hub like RichardErkhov (1.7PB and 25,000 repos) and mradermacher (6.1PB and 42,000 repos 🤯).
CAS throughput, meanwhile, has grown by an order of magnitude between the first and latest large-scale migrations:
When we began replacing LFS, we had two goals in mind:
Designing with our initial constraints and these goals allowed us to:
hf-xet before including it in huggingface_hub as a required dependencyInstead of waiting for all upload paths to become Xet-aware, forcing a hard cut-over, or pushing the community to adopt a specific workflow, we could begin migrating the Hub to Xet immediately with minimal user impact. In short, let teams keep their workflows and organically transition to Xet with infrastructure supporting the long-term goal of a unified storage system.
In January and February, we onboarded power users to provide feedback and pressure-test the infrastructure. To get community feedback, we launched a waitlist to preview Xet-enabled repositories. Soon after, Xet became the default for new users on the Hub.
We now support some of the largest creators on the Hub (Meta Llama, Google, OpenAI, and Qwen) while the community keeps working uninterrupted.
What's next?
Starting this month, we're bringing Xet to everyone. Watch for an email providing access to Xet and once you have it, update to the latest huggingface_hub (pip install -U huggingface_hub) to unlock faster transfers right away. This will also mean:
If you upload or download from the Hub using your browser or use Git, that's fine. Chunk-based support for both is coming soon. In the meantime use whichever workflow you already have; no restrictions.
Next up: open-sourcing the Xet protocol and the entire infrastructure stack. The future of storing and moving bytes that scale to AI workloads is on the Hub, and we're aiming to bring it to everyone.
If you have any questions, drop us a line in the comments 👇, open a discussion on the Xet team page.
此内容由惯性聚合(RSS阅读器)自动聚合整理，仅供阅读参考。 原文来自 — 版权归原作者所有。