How to Scale Diffusers Training With NeMo Automodel

The recent collaboration between NVIDIA and Hugging Face brings distributed training to the NVIDIA NeMo Automodel library. You can now fine-tune large-scale video and image diffusion models directly from the Diffusers library without manual checkpoint conversion. This integration handles the complex sharding and memory management required to scale from a single GPU to hundreds of cards.

Installation and Setup

The library is open-sourced under the Apache 2.0 license. You can install it directly via pip:

bash pip3 install nemo-automodel

For production deployments, NVIDIA provides a pre-built Docker container that includes all required dependencies and optimized kernels. You can pull the container using:

bash docker pull nvcr.io/nvidia/nemo-automodel:26.02.00

Drop-in Compatibility

NeMo Automodel subclasses standard AutoModel classes to provide a familiar interface. You can point the pretrained_model_name_or_path parameter directly at any Diffusers model ID on the Hugging Face Hub.

This approach lets you use your existing YAML recipes for training. You no longer need to rewrite model architectures or manually convert weights when moving from local testing to cluster-scale training.

Architecture and Performance

NeMo Automodel is a PyTorch DTensor-native training library. It uses FSDP2 (Fully Sharded Data Parallelism) for memory-efficient weight and gradient sharding. The workflow includes latent caching and multiresolution bucketing to handle variable aspect ratios in high-resolution image and video datasets.

To maximize throughput, the library integrates optimized kernels from TransformerEngine and DeepEP. DeepEP specifically accelerates training for Mixture-of-Experts (MoE) models. Benchmarks for MoE architectures show 3.4 to 3.7x higher training throughput and 29 to 32% less GPU memory usage compared to native Diffusers implementations.

Supported Models

The integration natively supports training via flow matching. Several prominent models are immediately supported:

• FLUX.1-dev: A 12B parameter text-to-image model.
• Wan 2.1: A 1.3B parameter text-to-video model.
• HunyuanVideo 1.5: A 13B parameter text-to-video model.

NVIDIA’s release demonstrates this capability with a full-transformer fine-tune of FLUX.1-dev using a 78-card Rider-Waite tarot dataset, processing high-resolution visual data at scale.

Tradeoffs and Limitations

While NeMo Automodel simplifies cluster execution, fine-tuning MoE models at the 12B and 13B parameter scale still requires substantial multi-node infrastructure. The network bandwidth between nodes will dictate your true throughput limits.

However, smaller architectures are highly accessible. The 1.3B parameter Wan 2.1 model is small enough to fit entirely on a single 40GB A100 GPU during fine-tuning.

To begin training your custom datasets, update your existing Diffusers configuration file to reference the NeMo Automodel engine and run your standard PyTorch distributed launch command.