Skip to content

[Dev] Skip broadcast for empty cu_seqlens tensors - #6320

Draft
lhb8125 wants to merge 1 commit into
NVIDIA:devfrom
lhb8125:denliu/skip-empty-cu-seqlens-broadcast
Draft

[Dev] Skip broadcast for empty cu_seqlens tensors#6320
lhb8125 wants to merge 1 commit into
NVIDIA:devfrom
lhb8125:denliu/skip-empty-cu-seqlens-broadcast

Conversation

@lhb8125

@lhb8125 lhb8125 commented Aug 6, 2026

Copy link
Copy Markdown
Contributor

Summary

  • skip the cu_seqlens payload broadcast when its announced size is zero
  • keep source and receiver control flow symmetric
  • avoid issuing a zero-count NCCL collective backed by a null data pointer
  • add source-rank and receiver-rank unit coverage

This is an independent correctness fix. It is not required by the full-CUDA-graph path, which returns before this broadcast, but it fixes no-CG/mock-data execution where the empty tensor can reach NCCL.

Validation

  • failing job before the fix: 2548853
  • passing job with the fix: 2549134
  • CHECK_ONLY=true BASE_REF=dev uv run --no-sync --project /home/scratch.hongbinl_sw/work/fsdp/agentic-mcore-dev --extra mcore-lint -- bash tools/autoformat.sh
  • PYTHONIOENCODING=utf-8 python3 tools/check_copyright.py megatron/training/utils/common_utils.py tests/unit_tests/training/test_common_utils.py
  • python3 -m py_compile megatron/training/utils/common_utils.py tests/unit_tests/training/test_common_utils.py

The local toolkit environment does not contain PyTorch, so import-dependent pytest/mypy execution is deferred to the Megatron CI container.

🤖 Generated with Codex

@copy-pr-bot

copy-pr-bot Bot commented Aug 6, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant