From NVIDIA Megatron-LM for visibility#18

Open

RaymondLi0 wants to merge 7059 commits into

bigcode-project:multi-query-attentionfrom

Collaborator

RaymondLi0 commented Jan 24, 2023

No description provided.

RaymondLi0 changed the base branch from multi-query-attention to before-merge

June 20, 2023 20:12

RaymondLi0 changed the base branch from before-merge to multi-query-attention

June 20, 2023 20:12

ko3n1g force-pushed the main branch from ad5a627 to cd71d4c Compare

February 18, 2026 09:18

DAISY-gh and others added 27 commits

April 24, 2026 03:14


          fix(checkpoint_inspector): allow empty --param-to-param-group-map-json (

f78ed05

#4403)

Co-authored-by: Antoni-Joan Solergibert <asolergibert@nvidia.com>


          Add the YARN support for hybrid_model (#4244)

4d6cdd5

Co-authored-by: Philip Petrakian <ppetrakian@nvidia.com>


          [training migration] Add container class for config dataclasses (#4227)

41ffa83

Signed-off-by: Maanu Grover <maanug@nvidia.com>
Co-authored-by: claude[bot] <209825114+claude[bot]@users.noreply.github.com>


          Inference: Fix broken functional tests on gitlab (#4454)

a1165fa


          SafeUnpickler class for safe pickle usage (#4319)

d4cacef

Signed-off-by: dimapihtar <dpykhtar@nvidia.com>


          get rid of weights_only=False (#4434)

109feda

Signed-off-by: dimapihtar <dpykhtar@nvidia.com>


          Inference | Per-block MoE routing storage for prefix caching (#4301)

64870c1

Co-authored-by: Siddharth Singh <sidsingh@nvidia.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>


          Add troubleshooting tip for 'access forbidden' (#4449)

017e684


          Fix checkpoint loading with rerun state machine (#4448)

3d7bcd3


          Add misc CUDA graph sugar to CudaGraphManager (#4425)

9b02206


          Inference: Add the embedding and output layer in the full_iteration_i…

35f76df

…nference cuda graph scope for hybrid models (#4440)


          Important bugfixes in local CG implementation that were leading to lo…

481efd0

…ss curve gaps for latent MoE models (#4433)

Signed-off-by: root <jiemingz@nvidia.com>


          fix: Replace polynomial rolling hash with SHA-256 for prefix caching (#…

e9abb6c

…4158)

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>


          feat(ckpt): expose validate_access_integrity knob on dist-ckpt load (#…

377af02

…4422)

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>


          Fix multivalidation (#3388)

241a5ca

Signed-off-by: rprenger <rprenger@nvidia.com>


          Add missing knob for reduce_scatter_with_fp32_accumulation (#4410)

f2dcd42

Signed-off-by: qiyuw <qiyuw@nvidia.com>
Co-authored-by: Antoni-Joan Solergibert <asolergibert@nvidia.com>


          Enable CUDA graphs for MTP inference (#4260)

03f4111

Signed-off-by: Keshav Santhanam <ksanthanam@nvidia.com>


          chore(beep boop 🤖): Bump (main) (2026-04-27)

1879dc2


          checkpoint integrity verification (#4305)

970c254

Signed-off-by: dimapihtar <dpykhtar@nvidia.com>


          Fix cache gating (#4455)

ebd70d3

Co-authored-by: Siddharth Singh <sidsingh@nvidia.com>


          [Main] Fix FusedAdam.use_decoupled_grad mis-set for Megatron-FSDP. (#…

…4427)

Signed-off-by: Cory Ye <cye@nvidia.com>


          add permute fusion into hybrid ep (#4089)

8c5cf05

Co-authored-by: root <root@eos0047.eos.clusters.nvidia.com>
Co-authored-by: root <root@eos0260.eos.clusters.nvidia.com>
Co-authored-by: Dennis(Zhenhuan) Liu <denliu@nvidia.com>


          Add ColocatedBridgeCommunicator for heterogeneous TP/DP MIMO training…

42e396e

… (NMFW-17) (#4368)

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>


          Fix incorrect bias display in extra_repr of Column/RowParallelLinear (#…

6fd6652

…4330)

Co-authored-by: mhh111 <mahonghao1@huawei.com>
Co-authored-by: Antoni-Joan Solergibert <asolergibert@nvidia.com>


          Fix assertion logic in combined_1f1b_schedule_for_interleaved_pipelin…

c8a4bfd

…ing (#4276)

Co-authored-by: Hanpeng Hu <haaanpeng@outlook.com>
Co-authored-by: Deepak Narayanan <deepakn94@gmail.com>
Co-authored-by: Antoni-Joan Solergibert <asolergibert@nvidia.com>


          ci: Fix event name reference in CI workflow condition for merge group (…

374fa85

…#4462)


          Add manual sync workflow from main to dev (#4165)

9c15290

Signed-off-by: oliver könig <okoenig@nvidia.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-authored-by: Philip Petrakian <>
Co-authored-by: oliver könig <okoenig@nvidia.com>

yanring and others added 30 commits

May 27, 2026 11:05


          [main] Refactor and Improve MoE Logginginit commit (#3431)

873678a

Co-authored-by: Robin Zhang <robinz@nvidia.com>
Co-authored-by: Dennis Liu <denliu@nvidia.com>
Co-authored-by: Philip Petrakian <ppetrakian@nvidia.com>
Co-authored-by: Shifang Xu <shifangx@nvidia.com>


          ci: validate release branch-rules (#4929)

4e52a9e

Signed-off-by: oliver könig <okoenig@nvidia.com>


          [Megatron-FSDP] Add conditional param.grad dereferencing logic to sup…

67b2f38

…port full-iteration (FWD-BWD) CUDA graphability. (#4663)

Signed-off-by: Cory Ye <cye@nvidia.com>


          test: restrict iter-time comparison to steady-state window (#5010)

71223d5

Signed-off-by: oliver könig <okoenig@nvidia.com>


          fix(test): pin eval-global-batch-size on 15b gb200 release configs (#…

a6d61fb

…5022)

Signed-off-by: oliver könig <okoenig@nvidia.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>


          [fix] Release MTP assertion when EP overlap with PP=1 (#4796)

286445c


          fix(test): widen iter-time steady-state window for short tests (#5023)

0cb4034

Signed-off-by: oliver könig <okoenig@nvidia.com>


          Perf fix (#4996)

805e24e

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>


          Add dev-feature preservation gate and change schedule (#4773)

457e3f7


          chore(test): remove orphan nemotron3_super_release_g200 dir (#5024)

146d171

Signed-off-by: oliver könig <okoenig@nvidia.com>


          Ignore Claude worktree directory (#5020)

7be1748


          Update copy-pr-bot.yaml [skip ci]

9d23a73


          ci: update CI workflow conditions for integration tests (#4658)

3e00820


          Add NVSkills CI request workflow (#5033)


          DDP wrap pg size fixes (#5006)

2a4f820

Signed-off-by: Maanu Grover <maanug@nvidia.com>


          fix(layer_wise): tag MTP-stage word_embeddings as is_embedding_or_out…

e0afe89

…put_parameter (#5034)


          Move LTS dependencies from pyproject.toml to Dockerfile.ci.lts (#4877)

4f2e7ce


          Use shared ModelOpt calibration loop on 0.45+ with 0.44 fallback fix (#…

88efdf7

…4881)

Signed-off-by: Keval Morabia <28916987+kevalmorabia97@users.noreply.github.com>
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>


          test(release): skip golden comparison on intermediate resume windows (#…

a5c2d1b

…5040)

Signed-off-by: oliver könig <okoenig@nvidia.com>


          [mimo] Thread position_ids through MimoModel for multimodal RoPE (#4938)

2ebca1e

Signed-off-by: Li Ding <liding@nvidia.com>


          build: Switch DSv3 on H100 to HybridEP (#5039)

e6ff4c3

Signed-off-by: oliver könig <okoenig@nvidia.com>


          Fix: Import unwrap_model from megatron.core.utils in modelopt examples (

ba9e0ea

#5045)

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>


          Simple and stable Inference APIs (#4697)

3c39d98

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>


          ci: Add notification step for MBridge downstream test results (#5028)

f63a46e


          Delete output tensor early (#4742)

f1b5516


          Support ScaledSReLU in TE grouped MLP fuser (#4859)

f8e3885

Co-authored-by: sraman <sraman@users.noreply.github.com>
Co-authored-by: Siddhartha Raman S <sraman@login-lyris01.lyris.clusters.nvidia.com>
Co-authored-by: a <a>
Co-authored-by: Charlie Truong <chtruong@nvidia.com>
Co-authored-by: sraman-rgb <270218152+sraman-rgb@users.noreply.github.com>


          Skip gradient updates when grad norm exceeds threshold (#3460)

Signed-off-by: Yi-Fu Wu <yifu.wu@gmail.com>
Co-authored-by: Gerald Shen <geshen@nvidia.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>


          Add 9 user skills (#5066)

6e0d14a

Signed-off-by: nvskills-svc-account <svc-nvskills-signing@nvidia.com>
Co-authored-by: nvskills-svc-account <svc-nvskills-signing@nvidia.com>


          test(nemotron): align nemotron3 super GB200 goldens with exit-interva…

54dc530

…l 4768 (#5069)

Signed-off-by: oliver könig <okoenig@nvidia.com>


          chore: Update transformer-engine dependency to version 2.16.0 (#4992)

6e091e1

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet