Skip to content

[NO REVIEW][atom/atom-vllm] fix MTP accept-rate drop at con>=256 on sparse-MLA (DSA) decode#1593

Closed
zejunchen-zejun wants to merge 2 commits into
hexwang/vllm_glm_mtpfrom
zejun/resolve_accept_ratio_drop_for_mtp_for_sparse_mla_when_con_256_0714
Closed

[NO REVIEW][atom/atom-vllm] fix MTP accept-rate drop at con>=256 on sparse-MLA (DSA) decode#1593
zejunchen-zejun wants to merge 2 commits into
hexwang/vllm_glm_mtpfrom
zejun/resolve_accept_ratio_drop_for_mtp_for_sparse_mla_when_con_256_0714

Conversation

@zejunchen-zejun

@zejunchen-zejun zejunchen-zejun commented Jul 14, 2026

Copy link
Copy Markdown
Collaborator

Root cause: at large MTP batches the sparse-indexer decode path issues one big deepgemm_fp8_paged_mqa_logits + top_k_per_row_decode launch whose later rows can retain stale sparse-KV indices in the persistent buffer (rows beyond the CU wave capacity), so requests past the cliff attend to wrong KV and their drafts get rejected -> acceptance halves (~50% -> ~25% at con256, cliff at seq 128).

so when specify max num seqs to the value which is smaller than 128, the accept ratio restores to the normal value, while for default 512, most of requests has low draft token accept ratio. Here is the histogram illustration.
image

Fix:

  • Chunk the sparse-indexer decode launches to <= get_num_sms()*wave_per_eu rows (new helper sparse_indexer_decode_rows_per_launch), so every row's logits/topk are actually computed. Applied to both native (deepseek_v2) and plugin paths.
  • triton_gather_kv_indices_sparse: use token_to_seq_idxs as the authoritative row count and assert topk/sparse_kv_indptr are large enough instead of silently truncating (which left stale rows in the persistent sparse-KV index buffer).
  • plugin: handle requires_padding by packing weights and unpacking topk indices.

atom-vllm

Configuration Position 1 Acceptance Rate Position 2 Acceptance Rate Position 3 Acceptance Rate Avg Draft Acceptance Rate
without fix 0.422 0.245 0.108 25.8%
with fix 0.721 0.494 0.235 48.3%

atom

Configuration Acceptance Rate Accepted 0 Tokens Accepted 1 Token Accepted 2 Tokens Accepted 3 Tokens
without fix 26.87% 52.30% 23.90% 14.70% 9.10%
with fix 53.53% 17.64% 27.65% 31.20% 23.51%

zejunchen-zejun and others added 2 commits July 14, 2026 15:59
…DSA) decode

Root cause: at large MTP batches the sparse-indexer decode path issues one big
deepgemm_fp8_paged_mqa_logits + top_k_per_row_decode launch whose later rows can
retain stale sparse-KV indices in the persistent buffer (rows beyond the CU wave
capacity), so requests past the cliff attend to wrong KV and their drafts get
rejected -> acceptance halves (~50% -> ~25% at con256, cliff at seq 128).

Fix:
- Chunk the sparse-indexer decode launches to <= get_num_sms()*wave_per_eu rows
  (new helper sparse_indexer_decode_rows_per_launch), so every row's logits/topk
  are actually computed. Applied to both native (deepseek_v2) and plugin paths.
- triton_gather_kv_indices_sparse: use token_to_seq_idxs as the authoritative
  row count and assert topk/sparse_kv_indptr are large enough instead of silently
  truncating (which left stale rows in the persistent sparse-KV index buffer).
- plugin: handle requires_padding by packing weights and unpacking topk indices.

Verified: con256 MTP3 acceptance restored to ~45%.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…native MTP padding crash)

The assert added in the accept-drop fix (topk_indices.shape[0] >=
token_to_seq_idxs.shape[0]) was too strict. On the native path token_to_seq_idxs
can be padded to a larger query layout than the indexer's actual rows (real
tokens < batch_size*max_seqlen_q under cudagraph padding), e.g. 12 real vs 16
padded -> AssertionError. The plugin path avoids it via explicit requires_padding
pack/unpack.

Restore the original min() intersection over
(token_to_seq_idxs, topk_indices, sparse_kv_indptr-1). The chunked indexer
launches (unchanged) are what actually fix the con>=256 stale-index acceptance
drop; min() here only drops padding rows, never real requests.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@zejunchen-zejun

Copy link
Copy Markdown
Collaborator Author

这个bug的现象是GLM5.2并发256的时候,atom和atom-vllm的draft token接受率从正常的50%掉到25-30%
一个batch的256个request只有前128个request的接受率正常,后128个request的接受率几乎是0,于是综合接受率腰斩

@zejunchen-zejun zejunchen-zejun changed the title [atom/atom-vllm] fix MTP accept-rate drop at con>=256 on sparse-MLA (DSA) decode [NO REVIEW][atom/atom-vllm] fix MTP accept-rate drop at con>=256 on sparse-MLA (DSA) decode Jul 14, 2026
@zejunchen-zejun

Copy link
Copy Markdown
Collaborator Author

real fix should be in aiter side: ROCm/aiter#4244

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant