HeyGen's 18B Avatar IV Hits 1.86x Speedup via Trillium TPUs

HeyGen has successfully ported its 18B-parameter Avatar IV model to Google Cloud’s latest AI infrastructure, detailing the transition in a joint engineering report. By moving from GPU-based environments to Trillium (v6e) TPUs, the company achieved a 1.86x speedup in generation latency. For developers building real-time video generation or conversational AI applications, the port demonstrates a viable path for deploying large-scale PyTorch models on XLA devices with minimal code rewrites.

Parallelism and Compilation

The transition relied heavily on torchax, a library designed to compile and execute PyTorch code on Google’s Accelerated Linear Algebra (XLA) backend. Because Avatar IV processes long-context video frames and relies on an 18B+ parameter architecture, the engineering team deployed the model across an eight-chip Trillium mesh.

To fit the model into TPU memory while maintaining high throughput for AI inference, the implementation utilized a multi-dimensional parallelism strategy:

• Fully Sharded Data Parallel (FSDP): Distributed the 18B parameters across the TPU mesh to optimize memory efficiency.
• Ulysses Sequence Parallelism: Handled the extensive memory demands of long-context video generation without bottlenecking individual chips.
• FlashAttention Integration: Leveraged TPU-optimized attention kernels to accelerate the forward pass.

Reducing Pipeline Bubbles

Achieving the 1.86x generation speedup required specific communication optimizations. The engineers pipelined exposed all-to-all collectives, which allowed them to overlap inter-chip communication with computation. By hiding this communication overhead, they minimized “bubble” time in the TPU pipeline, a critical requirement for maintaining the low latency needed in interactive video streaming.

Metric	Specification
Model Size	18B+ parameters
Hardware Base	Google Cloud Trillium (v6e) TPUs
Mesh Topology	8-chip Trillium mesh
Latency Improvement	1.86x speedup vs. previous GPU baseline
Compilation Target	XLA via torchax

Cost and API Implications

The optimization directly enables HeyGen’s “Streaming Avatar” API, which provides sub-second responses for interactive digital avatars. Beyond pure latency improvements, the shift to Trillium v6e yields a more favorable performance-per-dollar ratio for massive-scale video synthesis compared to traditional high-end GPU clusters. This allows the API to serve higher concurrent request volumes at lower infrastructure costs.

If you plan to run TPU workloads on Google Cloud for existing PyTorch architectures, the torchax framework provides a tested bridge. You should evaluate your model’s attention kernels for XLA compatibility and ensure your communication collectives can be pipelined to avoid idle silicon during long-context generation.