How to Speed Up MoE Fine-Tuning With NeMo AutoModel
Learn how to configure NVIDIA NeMo AutoModel in Transformers v5 to increase MoE training throughput and reduce GPU memory usage.
The new NVIDIA NeMo AutoModel integration allows you to use NVIDIA’s high-performance training stack directly within the Hugging Face Transformers ecosystem. Released in June 2026, this library injects Hopper and Blackwell optimizations into the standard API, delivering up to 3.7x higher training throughput and a 32% reduction in GPU memory usage for Mixture-of-Experts (MoE) architectures.
Previously, leveraging NVIDIA’s Megatron-Core required complex checkpoint conversions. NeMo AutoModel eliminates this conversion tax, allowing you to train frontier-scale models on local clusters using familiar PyTorch and Hugging Face patterns.
Installation and Basic Usage
NeMo AutoModel builds on the foundation of Transformers v5. It requires no extensive code rewrites. You inject the optimizations by adding a single import statement to your existing fine-tuning scripts.
When you import the nemo_automodel package, it automatically subclasses AutoModelForCausalLM. Calling from_pretrained() will then route the model weights through NVIDIA’s optimized TransformerEngine kernels. The official NeMo AutoModel repository provides the exact version requirements and installation commands for your specific CUDA environment.
Hardware and Parallelism Strategies
The library relies on a PyTorch DTensor-native design. It implements a standard PyTorch SPMD (Single Program Multiple Data) approach rather than relying on custom sharding implementations.
It supports four primary parallelism configurations for scaling across clusters:
- Expert Parallelism (EP): Shards expert weights across multiple GPUs to fit massive active parameter counts.
- Fully Sharded Data Parallelism v2 (FSDP2): Distributes model states across data parallel workers.
- Tensor Parallelism (TP) and Context Parallelism (CP): Splits individual tensors and attention contexts across devices.
- Pipeline Parallelism (PP): Partitions the model vertically across multiple nodes.
To handle the communication bottlenecks inherent in MoE training, the library utilizes DeepEP Fused All-to-All Dispatch. This specialized communication library overlaps token routing with computation, ensuring your GPUs are not stalled waiting for network transfers.
Supported Architectures
The integration provides immediate support for several major MoE families. The kernels are heavily optimized for NVIDIA DGX systems, specifically H100 (Hopper) and DGX Spark (Blackwell) devices.
| Model Family | Variant Examples | Notable Capabilities |
|---|---|---|
| NVIDIA Nemotron-3 Ultra | 550B (55B active) | Optimized for agentic reasoning and complex routing. |
| DeepSeek-V3 | 671B | Achieves up to 250 TFLOPs/sec/GPU with AutoModel. |
| Qwen3.5-MoE | 397B, 35B | Supported directly in the v26.04/v0.4.0 recipe list. |
| GLM-5 & MiniMax-M2.5 | Various | Full support in the latest release notes. |
Evaluating Your Workload
The choice between fine-tuning vs RAG often depends on the compute cost of the training run. By reducing memory overhead by up to 32%, NeMo AutoModel lowers the hardware threshold required to adapt large MoE models.
Review the latest release notes to confirm kernel compatibility with your specific DGX hardware generation before beginning a full-scale distributed training run.
Get Insanely Good at AI
The book for developers who want to understand how AI actually works. LLMs, prompt engineering, RAG, AI agents, and production systems.
Keep Reading
Open-Source MoK Megakernel Bypasses CPU Bottlenecks on NVL72
Cursor has open-sourced Mixture-of-Kittens, a fused compute and communication megakernel that increases MoE training throughput by up to 2.37x on NVIDIA GB200.
How to Scale Diffusers Training With NeMo Automodel
Learn how to fine-tune large diffusion models like FLUX.1-dev and Wan 2.1 across multiple GPUs using the NVIDIA NeMo Automodel library.
DiffusionGemma Shifts 26B Local Inference to Parallel Decoding
Google's 26B Mixture-of-Experts model abandons autoregressive generation for parallel text diffusion to hit 700 tokens per second on consumer GPUs.
Inkling 2 Mini: Thinking Machines loops 12 of 24 layers three times and matches Qwen3.8 27B with 5.4B active parameters
Thinking Machines released Inkling 2 Mini on November 20 under Apache 2.0: a 26.4B mixture-of-experts model in a 20GB NVFP4/MXFP8 checkpoint that runs on a 32GB Mac. It ties Qwen3.8 27B on the Artificial Analysis index at about a sixth of the cost per task, but it is verbose and weaker on multi-turn chat.
Thinking Machines' Inkling 2 Small jumps 18 points on Artificial Analysis, with Apache 2.0 weights still pending
The 594B-parameter, 17B-active model launched November 5 on Tinker with closed weights. It scores 31 on the Artificial Analysis index, but uses about four times the output tokens of its predecessor.