How to Profile Cloud TPUs With Google's Microbenchmark Suite

Google’s new open-source TPU microbenchmark suite gives developers granular performance metrics for evaluating Cloud TPU hardware. The tool replaces theoretical product specifications with empirical, real-world measurements across critical architectural components. You can use it to diagnose hardware bottlenecks, calculate precise memory bandwidth, and optimize your matrix operations across different accelerator generations.

Core Benchmark Components

The microbenchmark suite divides performance evaluation into five distinct functional categories. Measuring these individually allows you to isolate the specific constraints of your distributed workloads.

• Network: Measures the interconnect performance for collective communication primitives such as all-gather, all-reduce, reduce-scatter, and all-to-all. These operations run across the Inter-Chip Interconnect (ICI), and the suite outputs both total throughput (GB/s) and latency (seconds).
• Compute: Quantifies raw throughput using General Matrix Multiply (GEMM/Matmul) operations. The output provides TFLOPs and Model FLOPs Utilization (MFU), which tracks exactly how efficiently your workload leverages the Matrix Unit (MXU).
• High Bandwidth Memory (HBM): Assesses the effective bandwidth (GB/s) moving into and out of on-chip memory during large vector operations.
• Host Transfer: Evaluates the speed of data movement between the host CPU and the connected TPU accelerator.
• Attention: Generates specific performance telemetry for attention mechanisms, allowing you to optimize context handling in Transformer-based architectures.

Establishing a Roofline Model

The primary technical application of these benchmarks is constructing a Roofline model for your application. A Roofline model provides a visual performance framework to determine whether a specific workload is limited by memory bandwidth (a “memory wall”) or by computational throughput.

The framework calculates achievable throughput using the following formula:

Achievable Throughput = $\min(\text{Peak FLOPS}, \text{Arithmetic Intensity} \times \text{Memory Bandwidth})$

By plotting the results of the microbenchmarks onto this model, you can map out exactly where your application hits its ceiling. If the workload falls on the sloped part of the roofline, it is memory-bound. If it hits the flat upper limit, it is compute-bound. You can use this diagnostic data to adjust batch sizes, shard strategies, or memory access patterns before scaling up to larger pods.

Targeting Eighth-Generation Architectures

Google released the suite specifically to help teams optimize for both the prior Ironwood (TPU v7) architecture and the newly split eighth-generation TPU 8t and 8i architectures.

Because the eighth generation bifurcates the hardware into specialized chips, your benchmarks will look vastly different depending on the target:

• TPU 8t (Training): These chips are built for large-scale training runs, delivering up to 121 exaflops in a full 9,600-chip superpod. They include native FP4 support. Network benchmarks here will heavily emphasize ICI throughput across massive topologies.
• TPU 8i (Inference): Optimized for reasoning and agentic AI inference tasks. These chips feature 384 MB of on-chip SRAM (triple the previous generation) and use the Boardfly topology, which reduces network diameter by 56%. Network latency and HBM benchmarks are particularly critical when evaluating the 8i.

Framework Support and Execution

To prevent framework lock-in, the suite natively supports JAX, PyTorch (via TorchTPU), and vLLM. This means you do not need to rewrite your models in a specific Google-supported language to run the evaluation.

Because exact configuration parameters vary depending on your framework choice and cluster size, consult the official TPU benchmark documentation for the exact CLI flags and dependency requirements for your specific environment. When running TPU workloads in production, you should run these microbenchmarks as a baseline step before initiating any large-scale distributed training job.

Run the network benchmarks first to validate your pod networking topology, then move to the HBM and compute modules to profile individual chip performance.