DeepSeek open-sources DeepInfer, its inference engine

DeepSeek has published DeepInfer on GitHub (deepseek-ai/DeepInfer) under the Apache-2.0 licence. The README calls it the inference framework that powers DeepSeek's large-scale production services. The release was shared on Hacker News on October 20. The details below are from the repository README.

• Scope: an offline-inference release. The README lists what is not included but needed for a complete inference service: a KV cache system, prefill/decode disaggregation (described as mostly implemented in the engine) and load balancing across data-parallel replicas.
• Models and hardware: it supports DeepSeek-V4.1-Flash only (the default checkpoint is DeepSeek-V4.1-Flash-1017), on NVIDIA SM100 GPUs and Huawei Ascend 950 NPUs.
• Features listed: custom kernels (FlashMLA, DeepGEMM, DeepEP, TileKernels and others), attention data parallelism and MoE expert parallelism with load balancing, DSpark with confidence-scheduled verification, bounded replay for sliding-window attention, asynchronous scheduling, batch-invariant inference and deterministic sampling (NVIDIA only), and structured outputs (regex, JSON schema, EBNF).
• Benchmarks: the README shows prefill and decode charts, measured by DeepSeek on 4 GPUs for prefill and 64 for decode.
• The README says DeepInfer was originally built mainly on vLLM, with inspiration from SGLang and TensorRT-LLM.

The TLDR AI newsletter of October 21 links a piece titled "DeepSeek V4.1 Flash: Large-Scale Inference Deployment". TLDR's summary, not independently checked, says DeepInfer combines distributed inference, caching, load balancing and speculative decoding, and that DeepSeek processes 155 trillion tokens a day per cluster and earns $38 in revenue per GPU-hour.

By October 20 the repository had about 11,100 stars and 989 forks, and the Hacker News thread had 2,274 points and 873 comments.