[NO REVIEW][atom/atom-vllm] fix MTP accept-rate drop at con>=256 on sparse-MLA (DSA) decode#1593
Closed
zejunchen-zejun wants to merge 2 commits into
Conversation
…DSA) decode Root cause: at large MTP batches the sparse-indexer decode path issues one big deepgemm_fp8_paged_mqa_logits + top_k_per_row_decode launch whose later rows can retain stale sparse-KV indices in the persistent buffer (rows beyond the CU wave capacity), so requests past the cliff attend to wrong KV and their drafts get rejected -> acceptance halves (~50% -> ~25% at con256, cliff at seq 128). Fix: - Chunk the sparse-indexer decode launches to <= get_num_sms()*wave_per_eu rows (new helper sparse_indexer_decode_rows_per_launch), so every row's logits/topk are actually computed. Applied to both native (deepseek_v2) and plugin paths. - triton_gather_kv_indices_sparse: use token_to_seq_idxs as the authoritative row count and assert topk/sparse_kv_indptr are large enough instead of silently truncating (which left stale rows in the persistent sparse-KV index buffer). - plugin: handle requires_padding by packing weights and unpacking topk indices. Verified: con256 MTP3 acceptance restored to ~45%. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…native MTP padding crash) The assert added in the accept-drop fix (topk_indices.shape[0] >= token_to_seq_idxs.shape[0]) was too strict. On the native path token_to_seq_idxs can be padded to a larger query layout than the indexer's actual rows (real tokens < batch_size*max_seqlen_q under cudagraph padding), e.g. 12 real vs 16 padded -> AssertionError. The plugin path avoids it via explicit requires_padding pack/unpack. Restore the original min() intersection over (token_to_seq_idxs, topk_indices, sparse_kv_indptr-1). The chunked indexer launches (unchanged) are what actually fix the con>=256 stale-index acceptance drop; min() here only drops padding rows, never real requests. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Collaborator
Author
|
这个bug的现象是GLM5.2并发256的时候,atom和atom-vllm的draft token接受率从正常的50%掉到25-30% |
Collaborator
Author
|
real fix should be in aiter side: ROCm/aiter#4244 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Root cause: at large MTP batches the sparse-indexer decode path issues one big deepgemm_fp8_paged_mqa_logits + top_k_per_row_decode launch whose later rows can retain stale sparse-KV indices in the persistent buffer (rows beyond the CU wave capacity), so requests past the cliff attend to wrong KV and their drafts get rejected -> acceptance halves (~50% -> ~25% at con256, cliff at seq 128).
so when specify max num seqs to the value which is smaller than 128, the accept ratio restores to the normal value, while for default 512, most of requests has low draft token accept ratio. Here is the histogram illustration.

Fix:
atom-vllm
atom