SkyPilot Drops Cross-Cloud Egress Fees With Hugging Face Storage

Hugging Face and SkyPilot have integrated Hugging Face Storage as a native storage backend, allowing you to run zero-egress compute jobs across more than 20 cloud providers. Detailed in the Hugging Face integration announcement, this architecture decouples data residency from compute availability. You can store massive datasets and models on the Hugging Face Hub, then launch training or serving jobs on AWS, Google Cloud, Azure, or Lambda Cloud without incurring cross-cloud data transfer fees. Here is how to configure your workloads to use the new native integration.

Authentication and Setup

The integration relies on your existing Hugging Face credentials, ensuring seamless security across disparate compute environments. SkyPilot handles the orchestration and automatically passes your credentials to the underlying cloud instances.

To authenticate, you must set your standard Hugging Face token in your local environment before launching a task. SkyPilot reads the HF_TOKEN environment variable and provisions it securely within the target container or virtual machine. This means you do not need to manage separate cloud IAM roles or AWS access keys just to pull your data.

Mounting Read-Only Models and Datasets

The integration introduces a standardized hf:// URL scheme to the SkyPilot storage layer. You can use this scheme directly in your task definition YAML files to mount data.

To mount a read-only repository, use the syntax hf://<owner>/<model>. This applies to models, datasets, and Space repositories hosted on the Hugging Face Hub. When SkyPilot launches your job on a given cloud provider, it provisions the instance and mounts the Hugging Face repository directly into the file system.

Because Hugging Face Storage does not charge egress or CDN fees, the target cloud reads the data at no cost. This is particularly useful when you need to reduce LLM API costs by running your own open-weight models on cheaper, spot-instance GPU providers rather than transferring large model weights into an expensive primary cloud environment.

Writing Checkpoints to Hugging Face Buckets

For training and fine-tuning workloads, you need a destination to write intermediate checkpoints and final model weights. The integration supports read-write access to Hugging Face namespaces.

To configure a read-write mount, use the syntax hf://buckets/<namespace>/<bucket_name>. SkyPilot will mount this bucket, allowing your training scripts to output artifacts as if they were writing to a local disk. The background synchronization handles the transfer back to Hugging Face Storage.

Using this centralized storage backend simplifies multi-cloud strategies. For example, if GPUs become scarce on AWS, you can shift your training job to Lambda Cloud using the exact same SkyPilot YAML. Your training script will continue reading from and writing to the identical hf:// paths without requiring data migration.

Storage Deduplication with Xet Technology

Transferring large model checkpoints frequently can saturate network bandwidth and slow down iterative training loops. The Hugging Face Storage backend mitigates this using Xet technology.

Xet provides content-defined chunking and deduplication. When your script saves an incremental checkpoint or a model variant, the storage layer analyzes the data chunks and only transfers the specific chunks that have changed since the last sync. This drastically reduces the bandwidth and storage requirements for multi-epoch training runs, allowing the sync process to finish much faster than a standard whole-file upload.

Infrastructure Support and FUSE Configuration

Mounting remote storage often introduces security and permissions complications, particularly in containerized orchestration environments like Kubernetes.

To make this integration seamless across standard infrastructure, the engineering teams upstreamed hf-mount FUSE fixes. These fixes allow the storage backend to be mounted inside unprivileged containers. This represents the standard security posture for most Kubernetes clusters and Slurm environments, meaning you can deploy these zero-egress workloads on on-premise clusters just as easily as on public cloud providers.

Cost and Tradeoffs

Adopting this architecture replaces standard cloud storage costs with Hugging Face’s volume-based pricing. As of 2026, Hugging Face Private Storage tiers range from $8 to $12 per TB per month. Public repositories remain free, subject to community-impact guidelines.

Comparing this to traditional cloud egress fees—which typically hover around $23 per TB for AWS S3—the zero-egress model offers significant savings for data-heavy workloads. This compute-agnostic approach is already seeing production adoption, with companies utilizing it to bypass cloud lock-in. If you need to benchmark custom AI agent tools across different hardware profiles, storing the evaluation datasets on Hugging Face allows you to spin up instances anywhere without moving the data first.

While the elimination of egress fees and the inclusion of Xet deduplication are strong advantages, FUSE mounts inherently carry slight latency overhead compared to block storage natively attached to a compute instance. For workloads doing rapid, random reads of small files, you may need to adjust your data loading pipelines to pre-fetch sequentially.

Update your SkyPilot installation to the latest version to enable the hf:// syntax and begin migrating your storage backend.