Skip to content

feat(sglang): pool DeepSeek-V4 SWA KV cache - #400

Draft
shipiyouniao wants to merge 4 commits into
ovg-project:mainfrom
shipiyouniao:feat/sglang-dsv4-kv-pool-bridge
Draft

feat(sglang): pool DeepSeek-V4 SWA KV cache#400
shipiyouniao wants to merge 4 commits into
ovg-project:mainfrom
shipiyouniao:feat/sglang-dsv4-kv-pool-bridge

Conversation

@shipiyouniao

Copy link
Copy Markdown
Contributor

Summary

This is a stacked follow-up to #376 and part of #369. It adds an opt-in SGLang DeepSeek-V4 bridge that lets kvcached own the uncompressed SWA KV buffers and their physical page lifecycle.

This PR must not merge before #376. The current branch contains the #376 commits because the allocator compatibility and runtime-reservation accounting are hard dependencies. After #376 merges, I will rebase this PR so that only the SWA pooling commit remains.

Ownership boundary

kvcached owns:

  • the uncompressed DeepSeek-V4 SWA KV VMM buffers;
  • physical page allocation for the SWA pool on TP rank 0;
  • deterministic logical block IDs on the other TP ranks.

SGLang continues to own:

  • compressed C4/C128 KV pools;
  • indexer pools;
  • compressor and indexer state pools;
  • the model-specific store/compress execution path.

The full-token allocator used by SGLang's composite SWA allocator is logical-only after the bridge is active. It does not reserve a second physical KV payload.

Activation and failure behavior

The bridge is disabled by default and requires:

export KVCACHED_SGLANG_DSV4_KV_POOL_BRIDGE=1
export KVCACHED_CONTIGUOUS_LAYOUT=false

The per-layer layout is required because the current DSV4 kernels expect each layer's SWA buffer to be contiguous. With KVCACHED_REQUIRE=1, an incompatible runtime shape fails startup. Otherwise the adapter logs the incompatibility and leaves the native SGLang buffers in place.

The runtime-reservation patch from #376 excludes the native SWA reservation only after the bridge marks the SWA pool as successfully managed. Compressed/indexer/state reservations remain accounted as runtime-owned memory.

Validation

Focused regression tests:

21 passed

The full upstream pre-commit and typing matrix was reproduced on Linux and passed:

  • all pre-commit hooks, including ruff, format, codespell, isort, SPDX, clang-format, PyMarkdown, actionlint, and whitespace checks;
  • mypy for Python 3.9, 3.10, 3.11, 3.12, and 3.13.

The tests verify opt-in behavior, replacement of the real SWA buffers, preservation of runtime-owned side pools, physical allocation only on TP rank 0, logical allocation on peer TP ranks, reservation accounting, and idempotent patching.

This PR does not claim complete DeepSeek-V4 pool ownership or a performance improvement. GPU runtime validation remains required before the opt-in bridge is considered production-ready.

Closes no issue; #369 should remain open for the remaining DeepSeek-V4 pool families and runtime coverage.

@shipiyouniao
shipiyouniao force-pushed the feat/sglang-dsv4-kv-pool-bridge branch 2 times, most recently from 45a0105 to 90f8247 Compare July 24, 2026 05:08
@RixinLiu RixinLiu mentioned this pull request Aug 5, 2026
55 tasks
@shipiyouniao
shipiyouniao force-pushed the feat/sglang-dsv4-kv-pool-bridge branch 3 times, most recently from 0a8a044 to af6d1c7 Compare August 19, 2026 11:08
@shipiyouniao
shipiyouniao force-pushed the feat/sglang-dsv4-kv-pool-bridge branch 2 times, most recently from af6d1c7 to 91df367 Compare August 26, 2026 10:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant