DeepSeek Open-Sourced Its Production Inference Engine, and Huawei Gets Equal Billing

DeepSeek open-sourced DeepInfer on October 20, and the significance is double. First, this is not a research toy: it is the framework that powers DeepSeek’s own large-scale production serving, released under Apache 2.0 and already at roughly 9.5k stars within hours. Second, the hardware abstraction treats Huawei’s Ascend 950 NPUs as a first-class peer of NVIDIA’s SM100 GPUs: the build system takes DEEPINFER_ACCELERATOR=nvidia or =ascend, with CUDA 13 or CANN 9.2 as the respective runtimes. The framework ships with DeepSeek-V4.1-Flash support out of the box, defaulting to the 1017 checkpoint we covered on release day.

What is actually in the box

DeepInfer’s README is a tour of the techniques that make cheap frontier serving possible. The kernel layer bundles the stack DeepSeek has been publishing piecemeal for two years: FlashMLA, DeepGEMM, DeepEP, TileKernels, DeepSelect, and custom TileLang kernels. On top sit attention data parallelism with balanced context parallelism, MoE expert parallelism with EPLB load balancing, and asynchronous scheduling that overlaps CPU scheduling decisions with device execution, on by default. The speculative decoding design is the most distinctive piece: “confidence-scheduled verification” drafts up to five tokens and schedules verification based on model confidence, alongside “SWA bounded replay” to cut decoder prefill work. Deterministic sampling and batch-invariant inference are included but flagged NVIDIA-only, and structured outputs (regex, JSON schema, EBNF) are built in. The lineage matters for adopters: DeepInfer was built primarily on vLLM, with inspiration from SGLang and TensorRT-LLM, so migration paths are short.

What is deliberately missing, and the Huawei signal

The README is unusually honest about scope: KV cache management, prefill/decode disaggregation, and cross-replica load balancing, the components needed for a full production service, are not in this release. What ships is offline inference with the core large-scale serving features, meaning teams can benchmark and run batch workloads today but must build or wait for the serving layer. Performance claims are correspondingly restrained: Pareto charts rather than headline numbers, measured on 4 SM100 GPUs for prefill and 64 for decode, with a sweep script for reproduction. The Huawei integration is the geopolitical statement: a top Chinese lab’s production stack now builds identically for NVIDIA and domestic silicon, on the same day Intel pitched inference-specific hardware to the same buyers and Washington moved to ban DeepSeek from federal systems. The inference-engine layer is where hardware lock-in used to live, and DeepSeek just open-sourced theirs across both ecosystems.

Why this matters beyond the DeepSeek ecosystem

For serving engineers, DeepInfer is the first chance to run the exact stack behind one of the world’s busiest AI APIs, which makes its kernel choices (FlashMLA and DeepGEMM especially) and its confidence-scheduled speculative decoding a de facto reference architecture. For the open-source inference ecosystem, it sharpens an existing consolidation: vLLM, SGLang, and TensorRT-LLM already split most deployments, and a well-funded lab upstreaming its production fork changes the contribution calculus for all of them. And for the US policy conversation, the timing is pointed: hours after the federal ban, DeepSeek demonstrated that its software runs on Chinese silicon without translation. What to watch: whether the missing serving components follow (DeepSeek has a pattern of staged releases), whether Ascend 950 performance numbers appear in the project’s benchmark sweeps, and whether vLLM and SGLang teams upstream or diverge from the kernel work. The repo’s star velocity suggests the community has already voted on interest.