Cloudflare's GLM-5.3 Bug Leaked Other Customers' Prompts Through Stale KV Cache Pages

Cloudflare disclosed on October 26 that a KV-cache exposure in its Workers AI GLM-5.3 deployment let some requests receive responses generated while attending over other customers’ cached context: 155 responses affected between August 27 and September 9, all for prompts longer than 681,280 tokens, containing fragments of other customers’ prompts and generations made shortly before. Security researcher Gadi Evron of Knostic reported it via HackerOne on September 8; Cloudflare reproduced it in 25 minutes the next morning and had the model disabled within roughly 40 minutes of declaring an incident. No deliberate exploitation was found, and only GLM 5.3 was affected.

Two failures that had to line up perfectly

The exposure required a misconfiguration and an engine bug to align, which is why it slipped through. GLM 5.3 was deployed with a 1,048,576-token context window, but the KV cache pool on the prefill GPUs held only 681,280 tokens; Cloudflare knew about the mismatch and assumed SGLang would reject over-length prompts with an error. Their tests checked HTTP status codes, not output coherence. The second half was an SGLang bug in disaggregated prefill/decode mode: an over-length prompt was marked rejected and swapped for a one-token placeholder, prefill transferred just one KV page, but decode, trusting the “transfer complete” signal, generated a response attending over stale cache pages from prior requests, and returned HTTP 200. Every failure mode analysis lesson from distributed systems is in that sentence: the component that knew something was wrong did signal it, and the component that needed to listen trusted a completion flag instead.

The fix, and the hardening it triggered

Cloudflare’s remediation ran in layers. The immediate fix expanded the KV pool to roughly 2.04 million tokens, about twice the context length, and patched the engine so rejected requests retire at the admission queue before any KV transfer happens; the proposed fix went upstream as SGLang PR #38704, with related fixes landing in v0.5.20 and v0.5.21. The hardening went further: deferred KV page release on both prefill and decode roles, bounded Mooncake transfer batches, load balancer changes to avoid mid-transfer aborts, and mandatory pre-deployment checks comparing pool size against context length. Longer term, Cloudflare prototyped page-level ownership tagging at under 1% overhead, is contributing upstream KV checksums, and plans to formally verify the RDMA transfer protocol. Operators get a concrete checklist: measure KV pool size against context length on every node after any config change, run SGLang v0.5.22 or later, enable KV checksums and canaries where supported, and configure load balancers to avoid aborting transfers.

The pattern that should worry every inference provider

The structural lesson lands on the entire serving industry. Disaggregated prefill/decode architectures, KV-cache pools sized to expected traffic, and multi-tenant GPU sharing are the standard economics of inference in 2026; this incident shows the tenant-isolation boundary in that stack is enforced by convention, configuration, and engine trust, not by design. It is the inference-serving cousin of the sandbox escapes OpenAI disclosed last week: containment boundaries assumed to be structural turn out to be probabilistic. Three things to watch: whether other providers publish equivalent pool-versus-context audits, since Cloudflare’s own test suite missed this until a researcher looked; whether page-level ownership tagging becomes an upstream SGLang standard; and whether enterprise buyers add KV-isolation attestations to procurement, because the multi-tenant infrastructure serving every model is now demonstrably capable of crossing customer boundaries when two bugs line up, and 155 responses is what it looks like when they do.