How to Profile PyTorch Attention Kernels on A100 GPUs

Hugging Face has published the final installment of its educational series, Profiling in PyTorch (Part 3): Attention is all you profile. This guide breaks down the attention mechanism into its primitive components to demonstrate how they appear in a profiler trace. You will learn to identify compute-bound operations, evaluate Scaled Dot-Product Attention (SDPA) dispatching, and interpret hardware-level execution on NVIDIA A100 GPUs.

Understanding the Attention Trace

The PyTorch profiler exposes the fundamental building blocks of Transformer architectures by tracing individual GPU operations. When profiling a naive attention implementation, the trace records the explicit sequence of matrix multiplications (matmul(q, k.T)), scaling, masked_fill, softmax, and the final matmul(attn, v).

This granularity lets you isolate specific bottlenecks. You must categorize operations into two groups:

1. Compute-bound operations: Large GEMMs (General Matrix Multiplications) that saturate the GPU’s compute cores.
2. Memory-bound operations: Functions like softmax and masking that are constrained by memory bandwidth rather than processing power.

Analyzing Launch Overhead

A common pitfall in PyTorch profiling is misinterpreting CPU execution time. Traces often display wide, “fat” CPU execution bars that do not correspond to heavy computational workloads. These indicate launch overhead or Python-level bottlenecks rather than GPU execution issues. If your trace shows significant gaps between kernel executions on the GPU, you need to find GPU gaps caused by the CPU failing to queue kernels fast enough.

SDPA and Optimized Backends

PyTorch 2.13 introduced further optimizations for attention kernels. Instead of manually writing out the attention operations, you should use torch.nn.functional.scaled_dot_product_attention (SDPA). The profiler reveals how this built-in function dispatches to optimized backends.

Depending on your tensor shapes and hardware, SDPA automatically selects specialized implementations like FlashAttention or Memory-Efficient Attention. Tracing SDPA on an NVIDIA A100-SXM4-80GB GPU shows a significant reduction in memory bandwidth usage because these fused kernels avoid writing intermediate activation states to High Bandwidth Memory (HBM).

Profiling Methodology

Hugging Face recommends a “guess first, then look” approach to profiling. Before opening the trace, estimate which operations will dominate the runtime based on your tensor dimensions and how function calling works in your specific architecture.

When running custom Triton kernels for fused operations, verify backend awareness. The PyTorch dispatcher routes tasks to specialized libraries like cuDNN or cuBLAS. Check the profiler tables to confirm your kernel fused the intended operations rather than falling back to sequential cuBLAS calls. For reproducible environments, run your profiling scripts using Hugging Face Dev Mode with Spaces to isolate hardware-specific timing variations.