Skip to content

Add Nemotron 3.5 Lightning GB200 nightly test - #6312

Open
Phlip79 wants to merge 5 commits into
NVIDIA:mainfrom
Phlip79:philip/nemotron-3-5-lightning-gb200-nightly
Open

Add Nemotron 3.5 Lightning GB200 nightly test#6312
Phlip79 wants to merge 5 commits into
NVIDIA:mainfrom
Phlip79:philip/nemotron-3-5-lightning-gb200-nightly

Conversation

@Phlip79

@Phlip79 Phlip79 commented Aug 6, 2026

Copy link
Copy Markdown
Member
  • I, the PR author, have personally reviewed every line of this PR.

What does this PR do?

Adds a 2-node, 8-GPU Nemotron 3.5 Lightning GB200 nightly pretraining test based on Megatron Bridge's nemotron_3_5_lightning_pretrain_8k_config.

Details

  • Adds the 20-step, five-repeat TP1/PP1/CP1/EP8 functional test and GB200 golden values.
  • Registers the nightly-only dev / dgx_gb200 JET recipe using RP2 and SentencePiece data.
  • Sets segment: 2 explicitly so both Slurm nodes remain in the same NVLink domain if the scheduler default changes.
  • Extends functional metric extraction and thresholds to validate mtp_2 loss.
  • Uses native cross-entropy fusion as the compatibility override for current Megatron argument validation.

Validation

Tested here

Phlip79 added 4 commits August 6, 2026 04:07
Signed-off-by: Philip Petrakian <ppetrakian@nvidia.com>
Signed-off-by: Philip Petrakian <ppetrakian@nvidia.com>
Signed-off-by: Philip Petrakian <ppetrakian@nvidia.com>
Signed-off-by: Philip Petrakian <ppetrakian@nvidia.com>
@copy-pr-bot

copy-pr-bot Bot commented Aug 6, 2026

Copy link
Copy Markdown

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

Signed-off-by: Philip Petrakian <ppetrakian@nvidia.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant