Ai Engineering 2 min read

Open-Source MoK Megakernel Bypasses CPU Bottlenecks on NVL72

Cursor has open-sourced Mixture-of-Kittens, a fused compute and communication megakernel that increases MoE training throughput by up to 2.37x on NVIDIA GB200.

Cursor released Mixture-of-Kittens (MoK), an Apache 2.0-licensed Mixture-of-Experts megakernel built specifically for NVIDIA Blackwell NVL72 systems. By fusing computation and communication into a single deterministic kernel, MoK bypasses the CPU synchronization delays that typically stall large-scale MoE training.

Fusing Compute and Networking

Standard MoE architectures rely on external communication libraries like NCCL to manage inter-GPU routing. On GB200 and GB300 hardware, this creates bottlenecks when the integrated Grace CPUs lag behind the much faster Blackwell GPUs, leading to wasted GPU idle time.

MoK eliminates this dependency during the MoE phase. Built with pure CUDA and PTX using the ThunderKittens tile-primitive library, the kernel overlaps network transfers with computation at a configurable granularity. Instead of waiting for host-device handshakes, MoK uses ring token buffers to handle routing, effectively removing all CPU-GPU synchronization requirements within a 72-GPU NVLink domain.

The release represents a unified evolution of Cursor’s earlier warp decode optimizations, transitioning from custom MXFP8 work into a comprehensive megakernel.

Benchmark Results

Tested against leading public baselines, including implementations using NCCL, DeepEP, and TransformerEngine, MoK demonstrated substantial throughput improvements across both execution directions.

PrecisionForward Throughput GainBackward Pass Speedup
MXFP82.37x1.78x
BF161.92x1.58x

In practical application, Cursor reports that integrating MoK into their internal production stack yielded a 1.41x increase in end-to-end training throughput for their Composer and Grok models.

Hardware and Software Constraints

Because the kernel is heavily optimized for a specific network topology, deployment is highly constrained. MoK strictly targets SM100 and SM103 architectures, meaning it cannot be executed on older Hopper or Ampere hardware.

The required software stack includes Python 3.12+, PyTorch 2.10+, and CUDA Toolkit 13.0+.

If you manage training clusters for large-scale Mixture-of-Experts architectures, adopting a fused megakernel approach fundamentally alters your hardware utilization. By abstracting away host-side orchestration and executing the entire MoE layer directly on-device, MoK allows high-cost Blackwell capacity to spend significantly less time waiting on CPU synchronization and network routing.

Get Insanely Good at AI

Get Insanely Good at AI

The book for developers who want to understand how AI actually works. LLMs, prompt engineering, RAG, AI agents, and production systems.

Keep Reading

Ai Engineering

How to Speed Up MoE Fine-Tuning With NeMo AutoModel

Learn how to configure NVIDIA NeMo AutoModel in Transformers v5 to increase MoE training throughput and reduce GPU memory usage.

Ai Engineering

DiffusionGemma Shifts 26B Local Inference to Parallel Decoding

Google's 26B Mixture-of-Experts model abandons autoregressive generation for parallel text diffusion to hit 700 tokens per second on consumer GPUs.

Ai Engineering

Arcee Releases 400B Open-Source Trinity Model for Agents

The Trinity-Large-Thinking model offers a low-cost, open-source alternative for OpenClaw users following Anthropic's recent subscription policy changes.

Ai Engineering

Inkling 2 Mini: Thinking Machines loops 12 of 24 layers three times and matches Qwen3.8 27B with 5.4B active parameters

Thinking Machines released Inkling 2 Mini on November 20 under Apache 2.0: a 26.4B mixture-of-experts model in a 20GB NVFP4/MXFP8 checkpoint that runs on a 32GB Mac. It ties Qwen3.8 27B on the Artificial Analysis index at about a sixth of the cost per task, but it is verbose and weaker on multi-turn chat.

Ai Engineering

llama.cpp 1.0.0: semantic versioning for libllama and ggml, FP8 KV caches and a safer llama-server

llama.cpp and ggml shipped version 1.0.0 on November 13, three and a half years after the first commit. The release commits both libraries to major-version bumps for breaking C API and ABI changes, removes deprecated functions, and tightens llama-server's network defaults. Here is what changes for anyone who builds on it.