How to Configure Elastic Training in MaxText on TPUs

Google Cloud recently introduced elastic training for MaxText, allowing distributed AI workloads to survive hardware failures without crashing the entire cluster. By leveraging the Pathways execution backend, MaxText transforms TPU drops into catchable Python exceptions, recovering workloads in seconds rather than the typical 30 to 60 minutes. Here is how to configure elasticity for your next massive training run.

Understanding the Elastic Architecture

Traditional synchronous distributed training relies on an all-or-nothing approach. If a single accelerator fails in a 10,000-chip cluster, the entire job terminates. The new elastic framework decouples the training state from specific hardware nodes.

When a TPU v5p or TPU v6e node fails, the surviving nodes remain active. The control plane, managed by Google Kubernetes Engine (GKE) and the Kueue job orchestrator, automatically provisions a replacement or allows the job to continue on a reduced hardware slice. This self-healing approach maximizes your hardware goodput and significantly lowers the cost of training frontier models, an important consideration when scaling PyTorch training or JAX workloads.

Configuring MaxText Flags

Elastic training is now natively supported in the MaxText GitHub repository. To enable it, you must configure two specific flags in your training configuration:

• dynamic_loading: Set this to enable dynamic node provisioning. This allows the cluster to inject replacement hardware mid-training without restarting the global job.
• recovery: Enable this flag to activate the automated failure recovery protocol. It instructs Pathways to catch the hardware disconnect as a Python exception and hold the surviving nodes in a waiting state.

The MaxText documentation covers the full parameter list for fine-tuning the timeout thresholds and minimum cluster slice requirements during a hardware replacement.

Asynchronous Checkpointing

Elasticity requires the model to resume from the most recent state almost immediately. This is handled through Asynchronous Checkpointing via the Orbax library. By offloading the state save operations to background threads, Orbax minimizes the performance penalty of frequent checkpoints.

Configure Orbax in your MaxText run to save states frequently. If your Mean Time Between Failures (MTBF) is expected to be once per day, asynchronous checkpointing ensures you lose at most a few minutes of compute when the replacement node spins up. Similar mechanisms are used when you reduce LLM API costs by preventing full workload restarts.

Next Steps

Before deploying your model on a multi-thousand TPU cluster, simulate a node failure in a smaller GKE environment. Terminate a pod mid-training and verify that Kueue successfully provisions a replacement while the main job holds state.