Kog Inference Engine Reaches 3,000 Tokens/Sec on AMD MI300X

French AI infrastructure startup Kog has detailed the architecture behind its custom inference stack, demonstrating single-user speeds of up to 3,000 output tokens per second on standard datacenter GPUs. The Kog Inference Engine (KIE) challenges the industry shift toward specialized silicon by identifying standard software stacks, rather than the graphics processing units themselves, as the primary bottleneck for latency-sensitive workloads.

Since its tech preview in May 2026, the company has attracted over 200 enterprise leads looking to accelerate how AI agents work. Kog CEO Gaël Delalleau confirmed the platform is now scaling its optimization techniques to support mainstream Large Language Models (LLMs) and Mixture-of-Experts (MoE) architectures.

Hardware Agnosticism and Benchmarks

The KIE is optimized for single-request decoding speed. This metric is critical for autonomous agent loops that require dozens of internal reasoning, testing, and linting cycles before returning a final response. Kog claims a roughly 30x speedup over standard inference software like vLLM and SGLang for single-user workloads.

Hardware Configuration	Single-Request Decoding Speed
8x AMD MI300X	3,000 output tokens/second
8x NVIDIA H200	2,100 output tokens/second

Bypassing Standard Software Bottlenecks

The performance gains stem from three major architectural shifts that replace the default GPU software stack.

First, Kog built the Kog Communication Library (KCCL), a drop-in replacement for standard libraries like RCCL and NCCL. KCCL is specifically optimized to minimize latency in tensor parallelism across multiple cards.

Second, the engine uses Delayed Tensor Parallelism (DTP), a modified Transformer architecture. DTP hides communication overhead behind weight streaming, allowing the GPU to process tokens without waiting for synchronization pauses.

Finally, Kog relies on Monokernel Execution. The entire LLM decoding pass is implemented in a single persistent GPU kernel. This eliminates the microseconds lost to frequent kernel launches in standard AI inference pipelines.

The Kog Laneformer 2B Model

To demonstrate the engine’s capability, Kog released a custom instruction-tuned coding model designed for this architecture.

Kog Laneformer 2B is a 2.3-billion-parameter model that scores 45.1% on HumanEval+ and 51.6% on MBPP+ using greedy decoding. The model weights and inference code are available on the Hugging Face Hub under kogai/laneformer-2b-it.

If you build agentic workflows, the math around hardware requirements is shifting. Kog’s engine compresses what is typically a five-minute internal agent loop down to 10 seconds. For developers reducing API costs, extreme software co-design may offer a viable alternative to migrating to specialized application-specific integrated circuits for low-latency generation.