v26.4.0

What's Changed

• Merge Release/v26.3 back into main by @GeneDer in #733
• feat: add quantized tensor support and fix rmsnorm by @RuibinCheung in #735
• default NCCL_PXN_DISABLE to 0 to prevent failure of multi-node traini… by @amd-fuyuajin in #728
• feat(dashboard): unify weekly + monthly reports, fix responsiveness and redeploy by @WangLingxun in #739
• feat(torchtitan): SDMA copy-engine all-gather for FSDP by @lorri-rao in #740
• fix(runner): run direct launcher via array instead of eval by @WangLingxun in #744
• [Megatron-LM] fix: primus turbo attn memory footprint issue by @RuibinCheung in #745
• [Megatron-LM] fix: primus turbo grouped linear weight layout by @RuibinCheung in #743
• [Megatron-Bridge] Add support for Mamba SFT posttraining with Megatron-Bridge by @clairesonglee in #587
• feat(megatron): forward last-rank training_log to rank 0 for Megatron-Bridge by @WangLingxun in #747
• [Fix] pin down dependnecies for megatron by @GeneDer in #749
• feat(packaging): build & publish Primus wheel with bundled primus-cli by @HuangWei-95 in #750
• [Fix] unsupported moe_use_legacy_grouped_gemm arg in Mamba models by @GeneDer in #756
• [Megatron-LM] feat: add use_turbo_permute_padding to reduce d2h by @RuibinCheung in #742
• Add llama3.1 8b mxfp4 support by @vidushi8 in #748
• Add llama3.1 8b mxfp8 example by @vidushi8 in #746
• chore(skills): consolidate agent skills under skills/ with per-tool symlinks by @WangLingxun in #751
• feat(megatron):: support sft native by @wenxie-amd in #701
• chore(docker): bump base image rocm/primus v26.2 -> v26.3 by @WangLingxun in #737
• feat(packaging): third_party pip extras + fix dashboard refresh timing by @HuangWei-95 in #757
• feat(megatron): SDMA copy-engine all-gather for FSDP2 (use_torch_fsdp2) by @lorri-rao in #763
• fix PYTHONPATH for pip-installed primus by @amd-fuyuajin in #762
• [OOB Release] add jax v26.4 dockerfile by @GeneDer in #767
• Tuning agent, memory based benchmarking support plus fixes by @araina-amd in #755
• fix(v263): root-fix gfx942 hd128 backward crash in image + CI runtime by @WangLingxun in #764
• [projection] add unit tests; fix single-node DP and dense MoE-spec bugs by @WangLingxun in #760
• Add core unit tests (pipeline_parallel, utils) and report UT coverage in CI by @WangLingxun in #765
• fix(megatron): extend aiter DEEPBIND hd128 backward fix to gfx950 by @wenxie-amd in #768
• refactor(turbo): rename deprecated turbo flags and hard-assert on rem… by @wenxie-amd in #769
• Dev/update readme pip install by @GeneDer in #770
• docs: add CI status badges to README by @tanyaroosta in #771
• fix(torchtitan): correct MFU on MI350X/MI355X via peak FLOPS patch by @WangLingxun in #772
• fix(megatron): don't log from spawned async-checkpoint worker by @lorri-rao in #775
• [OOB Release] add primus v26.4 dockerfile by @GeneDer in #782

New Contributors

• @tanyaroosta made their first contribution in #771

Full Changelog: v26.3...v26.4.0

v26.3.1

Full Changelog: v26.3...v26.3.1

v26.2.1

Full Changelog: v26.2...v26.2.1

Primus release v26.3

What's Changed

• Pass missed HIPBLASLT env to docker by @Z-Y00 in #504
• container: pass HIPBLASLT_* env vars through to container by @Xiaoming-AMD in #506
• posttrain: enable megatron patches for megatron-bridge and fix intermittent framework parsing by @Xiaoming-AMD in #507
• tests: run megatron/torchtitan trainer UT via primus-cli direct (instead of run_pretrain.sh) by @Xiaoming-AMD in #508
• fix(yaml): remove deprecated turbo option from llama4 config by @RuibinCheung in #502
• Fixes RCCL all‑to‑all benchmark to use torch.distributed.group.WORLD only after init_process_group(). by @Vivicai1005 in #501
• Feat/fix mlflow multinode invalid metric log by @olehtika in #344
• fix(megatron): handle 2-element tensor in loss reducer by @dzautner in #340
• megatron: remove useless parameters from base configuration by @takaswie in #333
• ci(benchmark): add more model for single node by @HuangWei-95 in #509
• feat(primus-pipeline): add offload logic for PP input and attn OPs, fix some bugs by @ChengYao-amd in #511
• model summary tool by @wenxie-amd in #515
• fix(post-train): guard against HF datasets >= 3.0 incompatibility by @Xiaoming-AMD in #516
• feat(megatron-bridge): support backend parameter overrides by @Xiaoming-AMD in #518
• docs: update docs/examples to use ./primus-cli from repo root by @Xiaoming-AMD in #512
• update cicd runner by @Xiaoming-AMD in #526
• fix(megatron-bridge): add version constraint to turbo TE spec provider patch by @Xiaoming-AMD in #524
• Feature/backends/megatron mlflow env by @mvstrauss in #345
• ci(benchmark): split torch job to avoid timeout, add jax by @HuangWei-95 in #514
• feat(megatron-bridge): enable DEBUG logging for MI355X configs by @Xiaoming-AMD in #527
• feat(registry): add staged trainer registration with (backend, stage) key by @Xiaoming-AMD in #523
• fix(megatron): add TEGroupedMLP compatibility check for PrimusTurbo by @WangLingxun in #528
• [Megatron-LM] fix(mxfp4): get_fp4_context not be set correctly by @RuibinCheung in #522
• refactor(cli): improve subcommand loading, diagnostics, and patch version matching by @Xiaoming-AMD in #531
• fix(megatron): fix DelayedScaling incompatibility with save_original_… by @WangLingxun in #532
• Add HummingbirdXT backend integration by @limou102 in #533
• enable FP8 for DeepSeek v3 16B and fix Llama3.1 70B model path by @WangLingxun in #517
• refactor(megatron): add tokenizer override patch, move to new arch by @HuangWei-95 in #530
• ci(deterministic): add env for megatron ci test by @HuangWei-95 in #539
• Update Docker base image from v25.10 to v26.1 by @WangLingxun in #534
• Ci/test trainer/fix pipefail by @HuangWei-95 in #543
• feat(primus-pipeline): update PP simulation tools by @ChengYao-amd in #537
• fix:roolback turbo to 52337 by @Xiaoming-AMD in #541
• Refactor Training Runtime and Simplify Backend/Trainer Hierarchy by @Xiaoming-AMD in #536
• [Megatron-LM] fix(fp8): the static property of FP8GlobalStateManager not be set correctly by @RuibinCheung in #548
• docs: add backend extension guide and simplify pretrain hooks by @Xiaoming-AMD in #544
• ci(benchmark): fix megatron pattern for new arch by @HuangWei-95 in #546
• [Feature] Megatron-LM Llama 3.2 1B/3B Pretraining by @clairesonglee in #551
• Fix (ROCM): GPU count via GUID in rocm-smi by @WangLingxun in #547
• Multinode projection with different parallelization strategies when single node is benchmarked by @araina-amd in #492
• [Docs] & [Feature]: Add Post-Training Documentation and Update Qwen3_32B Configs for MI300X & MI355X by @kailashg26 in #556
• Fix/megatron/megatron fsdp patches by @jasainio in #553
• Fix/primus turbo/cpu init support by @jasainio in #552
• FIX: accumulate eval loss across all iterations by @sarthak-amd in #558
• doc(primus-pipeline): add primus-pipeline blog by @ChengYao-amd in #513
• Fix/preflight NUMA imbalance to mean uneven GPU distribution across nodes by @alexsu52 in #554
• update ainic bundle to 56. disable hipblaslt tuning by @wenxie-amd in #563
• feat(megatron): add average elapsed time metric by @HuangWei-95 in #561
• fix: normalize --key=value overrides by stripping leading dashes by @WangLingxun in #562
• chore(ci): update primus-turbo by @RuibinCheung in #550
• Expand projection.md with memory projection and performance details by @araina-amd in #576
• feat(megatron): migrate profiler args in core workflow by @HuangWei-95 in #573
• feat(megatron): migrate muon optimizer to core workflow by @HuangWei-95 in #574
• ci(benchmark): add multinode benchmark by @HuangWei-95 in #575
• Dev/fuyuajin/maxtext backend test by @amd-fuyuajin in #557
• refactor(config): unify override parsing and simplify runtime config normalization by @WangLingxun in #572
• fix(megatron): patch validate_args and add ROCM argument validation by @WangLingxun in #581
• enable moe_grouped_gemm in qwen by @wenxie-amd in #583
• fix(primus-pipeline): fix primus pipeline patch by @ChengYao-amd in #582
• fix(primus-pipeline): fix PRIMUS_TRAIN_RUNTIME=core issues by @ChengYao-amd in #586
• Add simulation backends and improve performance projection accuracy by @araina-amd in #560
• fix(run_pretrain): Enable NVTE_CK_USES_BWD_V3 for TE >= 2.6 by @Xiaoming-AMD in #596
• feat(config): add Qwen3 MoE model configs and update tests by @wenxie-amd in #590
• Fix/preflight handle local routes in route_to_master by @alexsu52 in #566
• Fix/preflight distinguish single-node distributed from multi-node by @alexsu52 in #565
• Primus docker release/v26.2 by @clairesonglee in #579
• refactor(maxtext): switch maxtext to register-patch workflow by @llying-001 in #601
• chore(megatron): bump version to core_v0.16.0 by @HuangWei-95 in #549
• fix(cli): pass --debug to python and include file:line in error msg by @HuangWei-95 in #592
• feat: Update Primus-Turbo by setting DeepEP CPU timeout via env. by @zhenhuang12 in #609
• fix(megatron): apply validate_args_modified path in base trainer by @WangLingxun in #588
• [Megatron-LM] fix: api compatible issue after mcore upgrade to v0.16.0 by @RuibinCheung in #617
• feat: add uccl-ep to docker built and add USING_UEP flag to run_pretrain and runner hook by @zhenhuang12 in #591
• refactor(maxtext): switch submodule maxtext to rocm/maxtext repo by @llying-001 in #615
• fix(hook): add 06_fix_aiter_asm_dir to resolve hipModuleLoad file-not-found by @WangLingxun in #621
• update primus-turbo version / dockerfile / CI by @xiaobochen-amd in #619
• fix(tests): validate training completion in UT and capture torchrun exit code by @WangLingxun in #620
• refactor(trainer): remove deprecated light-megatron backend path by @WangLingxun in #585
• [Feature] add support for gpt oss by @GeneDer in #616
• fix: use CUDA_DEVICE_MAX_CONNECTIONS from env if it is set, default to 1 by @zhenhuang12 in #623
• perf-projection: add pipeline scheduler comparison and fix plotext import by @araina-amd in https://githu...

v26.2.0rc1

Full Changelog: v26.2...v26.2.0rc1

Primus release v26.2

Docker version

This release tag is for building Docker v26.2.

What's Changed

• Pass missed HIPBLASLT env to docker by @Z-Y00 in #504
• container: pass HIPBLASLT_* env vars through to container by @Xiaoming-AMD in #506
• posttrain: enable megatron patches for megatron-bridge and fix intermittent framework parsing by @Xiaoming-AMD in #507
• tests: run megatron/torchtitan trainer UT via primus-cli direct (instead of run_pretrain.sh) by @Xiaoming-AMD in #508
• fix(yaml): remove deprecated turbo option from llama4 config by @RuibinCheung in #502
• Fixes RCCL all‑to‑all benchmark to use torch.distributed.group.WORLD only after init_process_group(). by @Vivicai1005 in #501
• Feat/fix mlflow multinode invalid metric log by @olehtika in #344
• fix(megatron): handle 2-element tensor in loss reducer by @dzautner in #340
• megatron: remove useless parameters from base configuration by @takaswie in #333
• ci(benchmark): add more model for single node by @HuangWei-95 in #509
• feat(primus-pipeline): add offload logic for PP input and attn OPs, fix some bugs by @ChengYao-amd in #511
• model summary tool by @wenxie-amd in #515
• fix(post-train): guard against HF datasets >= 3.0 incompatibility by @Xiaoming-AMD in #516
• feat(megatron-bridge): support backend parameter overrides by @Xiaoming-AMD in #518
• docs: update docs/examples to use ./primus-cli from repo root by @Xiaoming-AMD in #512
• update cicd runner by @Xiaoming-AMD in #526
• fix(megatron-bridge): add version constraint to turbo TE spec provider patch by @Xiaoming-AMD in #524
• Feature/backends/megatron mlflow env by @mvstrauss in #345
• ci(benchmark): split torch job to avoid timeout, add jax by @HuangWei-95 in #514
• feat(megatron-bridge): enable DEBUG logging for MI355X configs by @Xiaoming-AMD in #527
• feat(registry): add staged trainer registration with (backend, stage) key by @Xiaoming-AMD in #523
• fix(megatron): add TEGroupedMLP compatibility check for PrimusTurbo by @WangLingxun in #528
• [Megatron-LM] fix(mxfp4): get_fp4_context not be set correctly by @RuibinCheung in #522
• refactor(cli): improve subcommand loading, diagnostics, and patch version matching by @Xiaoming-AMD in #531
• fix(megatron): fix DelayedScaling incompatibility with save_original_… by @WangLingxun in #532
• Add HummingbirdXT backend integration by @limou102 in #533
• enable FP8 for DeepSeek v3 16B and fix Llama3.1 70B model path by @WangLingxun in #517
• refactor(megatron): add tokenizer override patch, move to new arch by @HuangWei-95 in #530
• ci(deterministic): add env for megatron ci test by @HuangWei-95 in #539
• Update Docker base image from v25.10 to v26.1 by @WangLingxun in #534
• Ci/test trainer/fix pipefail by @HuangWei-95 in #543
• feat(primus-pipeline): update PP simulation tools by @ChengYao-amd in #537
• fix:roolback turbo to 52337 by @Xiaoming-AMD in #541
• Refactor Training Runtime and Simplify Backend/Trainer Hierarchy by @Xiaoming-AMD in #536
• [Megatron-LM] fix(fp8): the static property of FP8GlobalStateManager not be set correctly by @RuibinCheung in #548
• docs: add backend extension guide and simplify pretrain hooks by @Xiaoming-AMD in #544
• ci(benchmark): fix megatron pattern for new arch by @HuangWei-95 in #546
• [Feature] Megatron-LM Llama 3.2 1B/3B Pretraining by @clairesonglee in #551
• Fix (ROCM): GPU count via GUID in rocm-smi by @WangLingxun in #547
• Multinode projection with different parallelization strategies when single node is benchmarked by @araina-amd in #492
• [Docs] & [Feature]: Add Post-Training Documentation and Update Qwen3_32B Configs for MI300X & MI355X by @kailashg26 in #556
• Fix/megatron/megatron fsdp patches by @jasainio in #553
• Fix/primus turbo/cpu init support by @jasainio in #552
• FIX: accumulate eval loss across all iterations by @sarthak-amd in #558
• doc(primus-pipeline): add primus-pipeline blog by @ChengYao-amd in #513
• Fix/preflight NUMA imbalance to mean uneven GPU distribution across nodes by @alexsu52 in #554
• [Primus] Fix MoE MLA issue from hybrid models branch merge by @clairesonglee in #564
• update ainic bundle to 56. disable hipblaslt tuning by @wenxie-amd in #563
• feat(megatron): add average elapsed time metric by @HuangWei-95 in #561
• fix: normalize --key=value overrides by stripping leading dashes by @WangLingxun in #562
• Expand projection.md with memory projection and performance details. by @araina-amd in #578
• chore(ci): update primus-turbo by @RuibinCheung in #550
• Expand projection.md with memory projection and performance details by @araina-amd in #576
• feat(megatron): migrate profiler args in core workflow by @HuangWei-95 in #573
• feat(megatron): migrate muon optimizer to core workflow by @HuangWei-95 in #574
• ci(benchmark): add multinode benchmark by @HuangWei-95 in #575
• Dev/fuyuajin/maxtext backend test by @amd-fuyuajin in #557
• refactor(config): unify override parsing and simplify runtime config normalization by @WangLingxun in #572
• fix(megatron): patch validate_args and add ROCM argument validation by @WangLingxun in #581
• enable moe_grouped_gemm in qwen by @wenxie-amd in #583
• fix(primus-pipeline): fix primus pipeline patch by @ChengYao-amd in #582
• fix(primus-pipeline): fix PRIMUS_TRAIN_RUNTIME=core issues by @ChengYao-amd in #586
• Add simulation backends and improve performance projection accuracy by @araina-amd in #560
• fix(run_pretrain): Enable NVTE_CK_USES_BWD_V3 for TE >= 2.6 by @Xiaoming-AMD in #596
• feat(config): add Qwen3 MoE model configs and update tests by @wenxie-amd in #590
• Fix/preflight handle local routes in route_to_master by @alexsu52 in #566
• Fix/preflight distinguish single-node distributed from multi-node by @alexsu52 in #565
• Primus docker release/v26.2 by @clairesonglee in #579
• refactor(maxtext): switch maxtext to register-patch workflow by @llying-001 in #601
• [Megatron-LM] Update Mamba model tokenizer by @clairesonglee in #603
• chore(megatron): bump version to core_v0.16.0 by @HuangWei-95 in #549
• fix(cli): pass --debug to python and include file:line in error msg by @HuangWei-95 in #592
• feat: Update Primus-Turbo by setting DeepEP CPU timeout via env. by @zhenhuang12 in #609
• fix(megatron): apply validate_args_modified path in base trainer by @WangLingxun in #588
• [Megatron-LM] fix: api compatible issue after mcore upgrade to v0.16.0 by @RuibinCheung in #617
• feat: add uccl-ep to docker built and add USING_UEP flag to run_pretrain and runner hook by @zhenhuang12 in #591
• refactor(maxtext): switch submodule maxtext to rocm/maxtext repo by @llying-001 in #615
• fix(hook): add 06_fix_aiter_asm_dir to resolve hipModuleLoad file-not-found by @WangLingxun in #621
• update primus-turbo version / dockerfile / CI by @xiaobochen-amd in #619
• fix(tests): validate training completion in UT and capture torchrun exit code by @WangLingxun in #620
• refactor(trainer): remove deprec...

v0.7.0

Docker Version

This release tag is for building Docker v26.1.

What's Changed

• [Primus performance projection] Initial version of pipeline simulation using measured layer-wise latencies. by @yuankaichen-amd in #362
• [Megatron][feat] Add deterministic traning support for megatron-lm backend by @RuibinCheung in #376
• refactor(megatron): Add Megatron MoE patch modules by @Xiaoming-AMD in #443
• refactor(megatron): Add Transformer Engine patches module by @Xiaoming-AMD in #445
• refactor(megatron): Add Primus Turbo patches module by @Xiaoming-AMD in #446
• refactpr(megatron): add zbpp patches for Megatron zero-bubble pipeline by @Xiaoming-AMD in #447
• feat(megatron): add FP8 patches for Megatron fp8 context by @Xiaoming-AMD in #448
• feature(megatron): Megatron PP patches into parallelism package by @Xiaoming-AMD in #449
• feature(megatron): Move MLA attention patch into Megatron patch system by @Xiaoming-AMD in #450
• feature(megatron): Add recompute_layer patches to Megatron patch system by @Xiaoming-AMD in #451
• feature(megatron): Add Torch FSDP2 patch to Megatron patch system by @Xiaoming-AMD in #452
• feature: Integrate Primus runtime patch system with Megatron backend by @Xiaoming-AMD in #453
• Migrate Megatron patch logic to Primus patch framework by @Xiaoming-AMD in #442
• refactor: integrate preflight tool into Primus CLI by @Xiaoming-AMD in #372
• docs: align Megatron example with current config by @Xiaoming-AMD in #454
• RCCL benchmarking update: FSDP and rccl-test command generation by @lorri-rao in #302
• fix(preflight): lazy-load preflight runner to avoid JAX UT import failure by @Xiaoming-AMD in #457
• feat(cli): let pretrain hooks return extra args via stdout by @Xiaoming-AMD in #455
• feat(primus-pipeline): support combined 1f1b for zbv-formatted, fix s… by @ChengYao-amd in #456
• refactor: move TorchTitan trainer patches into backend patch system by @Xiaoming-AMD in #459
• fix up evaluate loss for megatron by @llying-001 in #458
• feat(primus-pipe): optimize communication overlap by @ChengYao-amd in #461
• Align Primus TFLOPS computation with Megatron default definition by @Xiaoming-AMD in #462
• Megatron: align training flow with Primus patch architecture by @Xiaoming-AMD in #460
• Fix and modernize GEMM benchmark script by @Xiaoming-AMD in #465
• CI: Temporarily disable ainic docker image build by @Xiaoming-AMD in #468
• feat: add optional torchft dependencies for fault tolerance support by @zhanglei-amd in #467
• NCCL_IB_GID_INDEX on base_env.sh by @agelman-dn in #466
• feat: allow container name configuration and add user to default name by @olehtika in #347
• feat(runner): improve primus-cli env, hooks, and patch handling by @Xiaoming-AMD in #469
• benchmark: enhance RCCL microbench reporting and CLI integration by @Xiaoming-AMD in #290
• update llama & grok config by @JohnQinAMD in #472
• [Megatron-LM] feat(mxfp4): support mxfp4 in megatron-lm backend by @RuibinCheung in #470
• fix training issue on NVIDIA GPUs by @wenxie-amd in #473
• feat(torchtitan): integrate TorchTitan into Primus new architecture by @Xiaoming-AMD in #463
• Maximize batch sizes for DeepSeek V3 on MI355X by @clairesonglee in #366
• fix(torchtitan): adjust DeepSeek V3 16B batch size to 8 by @Xiaoming-AMD in #474
• Fix Megatron Patch System Initialization Flow by @Xiaoming-AMD in #475
• Improve Slurm CLI entry env setup and logging by @Xiaoming-AMD in #476
• Switch TorchTitan pretrain to core runtime by default and unify TorchTitan patch logging by @Xiaoming-AMD in #477
• Add MaxText Support to primus-cli by @Xiaoming-AMD in #478
• fix tp issue by @wenxie-amd in #479
• Pass RCCL_* and IONIC_* environment variables by @agelman-dn in #482
• feat(moun): add moun optimizer config by @ChengYao-amd in #483
• Add CLI-based example scripts for training workflow by @Xiaoming-AMD in #480
• megatron(turbo): rely on patch system for missing imports in spec provider by @Xiaoming-AMD in #485
• [Fix] fix quant config arg name by @GeneDer in #488
• Docs: add Benchmark Suite documentation by @Xiaoming-AMD in #491
• benchmark: remove legacy kernel/gemm script by @Xiaoming-AMD in #493
• feat(benchmark): add attention suite; remove legacy kernel attention by @Xiaoming-AMD in #494
• Preflight refactor: unified info/perf flow, simplified flags, richer host+GPU+network report by @Xiaoming-AMD in #490
• megatron: improve compatibility with Megatron-LM v0.10.0+ (args & pretrain API) by @Xiaoming-AMD in #496
• real ip address by @zhanglei-amd in #481
• Bugfix/titan patch by @wenxie-amd in #497
• feat: add build_uccl hook and install rocSHMEM in Dockerfile by @zhenhuang12 in #487
• ci(benchmark): add daily benchmark script by @HuangWei-95 in #489
• Add FP8 Support to GEMM Benchmarks by @Xiaoming-AMD in #499
• Post-Training Framework(Megatron-Bridge) Support by @Xiaoming-AMD in #500
• Enable turbo classic attention by @clairesonglee in #503

New Contributors

• @zhanglei-amd made their first contribution in #467
• @agelman-dn made their first contribution in #466

Full Changelog: v0.6.0...v0.7.0

v0.6.0

Docker Version

This release tag is for building Docker v25.11.

What's Changed

• fix(config): update TorchTitan fp8 configs for new converter and training schema by @Xiaoming-AMD in #236
• feat(torchtitan): auto-install nightly torch based on ROCm version before dataset preparation by @Xiaoming-AMD in #237
• feat(torchtitan): add Qwen3 model configs (0.6B / 1.7B / 32B) by @Xiaoming-AMD in #238
• feat(megatron): Primus-Megatron use PrimusTurboSpecProvider as models backends. by @zhenhuang12 in #231
• Refactor: Organize training configs by GPU architecture (MI300 / MI355) by @Xiaoming-AMD in #240
• fix(compat): add torch API compatibility patches for Titan imports on ROCm 7.0 by @Xiaoming-AMD in #242
• [titan] feat(config): add llama3 & llama3.3 config by @RuibinCheung in #239
• feat(zerobubble): zerobubble add lagacy group gemm & te backend by @ChengYao-amd in #241
• Support Megatron's a2a and deepep overlap in pipeline by @yuankaichen-amd in #244
• feat(torchtitan): add DeepSeek-V3 16B & 671B configs for MI300X and MI355X by @Xiaoming-AMD in #243
• feat(deepep): support fully syncfree moe stage3. by @zhenhuang12 in #232
• fix: parse moe_layer_freq from string to list in Megatron config adapter by @Xiaoming-AMD in #245
• Build Primus Docker (Private) by @wenxie-amd in #247
• chore: update turbo by @xiaobochen-amd in #248
• feat(benchmark/gemm): add base, dense, and deepseek GEMM benchmarks by @Xiaoming-AMD in #226
• upgrade torch to 2.10.0 nightly by @wenxie-amd in #249
• Fix: Align nn.Embedding output with AMP autocast precision by @Xiaoming-AMD in #251
• fix: fix torchtitan traning issue in TurboAttention by @kyle-256 in #253
• chore: llama-3.1-8b enable turbo by @xiaobochen-amd in #250
• feat(torchtitan): enable dynamic model parameter override via CLI by @Xiaoming-AMD in #254
• updage docker image to nightly; add turbo fa env by @wenxie-amd in #255
• test(torchtitan): add model unit tests for TorchTitan backend by @Xiaoming-AMD in #256
• Feature: Add Mock HuggingFace Dataset Support for TorchTitan by @Xiaoming-AMD in #260
• fix(deepep): fix moe overlap error with sync-free moe. by @zhenhuang12 in #258
• fix(tp-overlap): adapt transformer_engine 2.4 for Megatron backend by @zhenhuang12 in #259
• feat(zero-bubble): reorder comm-nodes for batch-p2p by @ChengYao-amd in #257
• refactor(torchtitan): rollback Titan to 99c0cb2(20250907) and stabilize trainer UTs by @Xiaoming-AMD in #262
• feat(turbo): Add turbo RMSNorm patch by @ChengYao-amd in #263
• fix: Disable double DDP construction inside build_model() via runtime patch by @Xiaoming-AMD in #264
• fix(light-megatron): resolve config parsing and adapter compatibility issues by @Xiaoming-AMD in #265
• Support torch_dist async checkpoint for latest Megatron-LM by @limou102 in #267
• New envs: HSA_KERNARG_POOL_SIZE, ENABLE_NUMA_BINDING by @wenxie-amd in #268
• feat: megatron support turbo fp8 grouped gemm by @xiaobochen-amd in #261
• feat(cli): add dataset preparation hooks for train/pretrain workflows by @Xiaoming-AMD in #270
• strided allgather benchmark by @wenxie-amd in #271
• disable cross entropy flags to avoid convergence loss divergence by @clairesonglee in #269
• [Fix] import MXLinear from Primus Turbo by @GeneDer in #272
• add public primus-safe link in readme by @wenxie-amd in #274
• fix readme's typo error by @wenxie-amd in #275
• Add memory projection cli by @yuankaichen-amd in #273
• feat(CLI): Add Runner Library and Test Suite by @Xiaoming-AMD in #276
• feat(runner): add patch execution system with comprehensive test suite by @Xiaoming-AMD in #279
• feat: Support wgrad in MoE overlap by @yuankaichen-amd in #278
• feature(cli): refactor environment configuration with layered design by @Xiaoming-AMD in #280
• refactor: Runner CLI Refactoring and Optimization by @Xiaoming-AMD in #281
• Documentation Reorganization and Structure Improvements by @Xiaoming-AMD in #283
• docs: reorganize backend patch notes + link docs by @Xiaoming-AMD in #284
• feature(cli): cli auto-discover subcommands and add tests by @Xiaoming-AMD in #285
• specify recompute_layer_ids by @llying-001 in #286
• fix(zerobubble): fix zerobubble pp warmup error by @ChengYao-amd in #287
• fix(token_dispatcher): fix deep_ep internode_combine hang when enable sync-free stage 2. by @zhenhuang12 in #289
• fix loading data bug when using vpp by @lhzhang333 in #277
• benchmark: emit GEMM reports as markdown by @Xiaoming-AMD in #288
• support maxtext backend by @llying-001 in #291
• bench: fix dense gemm pre-commit issues by @Xiaoming-AMD in #295
• feat(IRLens): open-source the IRLens tool by @yihuang-amd in #296
• fix errors when patch_moe_overlap is set and distribute Wgrads to overlap with different comms by @yuankaichen-amd in #294
• deprecate decoder_pipeline_manual_split_list by @lhzhang333 in #293
• feat(config): make Megatron workspace path configurable via env var by @poznano-amd in #298
• feat(IRLens): fix nested sub-computations called via call() operations by @yihuang-amd in #300
• chore: update latest turbo by @xiaobochen-amd in #301
• MoE package - version1 by @wenxie-amd in #282
• fix multi node aiter build bug by @wenxie-amd in #303
• maxtext: add xla_dump_hlo switch, update requirement and dpsk_v2_light config by @llying-001 in #304
• primus ainic docker image by @wenxie-amd in #308
• fix:megatron trainer import save_checkpoint_and_time by @olehtika in #307
• Refactor(config): Unify config inheritance and enhance Megatron args handling by @Xiaoming-AMD in #312
• feature(core): Add Patch Framework for Backend- and Version-Aware Patch Handling by @Xiaoming-AMD in #314
• Print global max mem usage by @wenxie-amd in #317
• backend(maxtext): support custom model condifg and update model args via cli by @llying-001 in #315
• feature(core): Add BackendAdapter unit tests and improve trainer creation workflow stability by @Xiaoming-AMD in #318
• feat(IRLens): use different variable name for nested loops by @yihuang-amd in #319
• feature(core): add BackendRegistry with lazy loading and comprehensive tests by @Xiaoming-AMD in #320
• feature(backends): enhance MegatronArgBuilder argument construction by @Xiaoming-AMD in #322
• feature(core): add TrainerComponent and unify trainer patch workflow by @Xiaoming-AMD in #323
• feat(backends): introduce Megatron backend trainers and pretrain wiring tests by @Xiaoming-AMD in #324
• feat(backends): Add Megatron backend adapter and wiring tests by @Xiaoming-AMD in #325
• fix(benchmark): fix gemm_bench.py duplicate and indent by @olehtika in #306
• feat(core): add unified train runtime orchestrator and optional CLI entry by @Xiaoming-AMD in #327
• backends/megatron: add args patches for paths, logging and wandb by @Xiaoming-AMD in #328
• backends/megatron: add checkpoint patch and platform_config fallback by @Xiaoming-AMD in #329
• backend...

v0.5.0

What's Changed

• fix(config): update TorchTitan fp8 configs for new converter and training schema by @Xiaoming-AMD in #236
• feat(torchtitan): auto-install nightly torch based on ROCm version before dataset preparation by @Xiaoming-AMD in #237
• feat(torchtitan): add Qwen3 model configs (0.6B / 1.7B / 32B) by @Xiaoming-AMD in #238
• feat(megatron): Primus-Megatron use PrimusTurboSpecProvider as models backends. by @zhenhuang12 in #231
• Refactor: Organize training configs by GPU architecture (MI300 / MI355) by @Xiaoming-AMD in #240
• fix(compat): add torch API compatibility patches for Titan imports on ROCm 7.0 by @Xiaoming-AMD in #242
• [titan] feat(config): add llama3 & llama3.3 config by @RuibinCheung in #239
• feat(zerobubble): zerobubble add lagacy group gemm & te backend by @ChengYao-amd in #241
• Support Megatron's a2a and deepep overlap in pipeline by @yuankaichen-amd in #244
• feat(torchtitan): add DeepSeek-V3 16B & 671B configs for MI300X and MI355X by @Xiaoming-AMD in #243
• feat(deepep): support fully syncfree moe stage3. by @zhenhuang12 in #232
• fix: parse moe_layer_freq from string to list in Megatron config adapter by @Xiaoming-AMD in #245
• Build Primus Docker (Private) by @wenxie-amd in #247
• chore: update turbo by @xiaobochen-amd in #248
• feat(benchmark/gemm): add base, dense, and deepseek GEMM benchmarks by @Xiaoming-AMD in #226
• upgrade torch to 2.10.0 nightly by @wenxie-amd in #249
• Fix: Align nn.Embedding output with AMP autocast precision by @Xiaoming-AMD in #251
• fix: fix torchtitan traning issue in TurboAttention by @kyle-256 in #253
• chore: llama-3.1-8b enable turbo by @xiaobochen-amd in #250
• feat(torchtitan): enable dynamic model parameter override via CLI by @Xiaoming-AMD in #254
• updage docker image to nightly; add turbo fa env by @wenxie-amd in #255
• test(torchtitan): add model unit tests for TorchTitan backend by @Xiaoming-AMD in #256
• Feature: Add Mock HuggingFace Dataset Support for TorchTitan by @Xiaoming-AMD in #260
• fix(deepep): fix moe overlap error with sync-free moe. by @zhenhuang12 in #258
• fix(tp-overlap): adapt transformer_engine 2.4 for Megatron backend by @zhenhuang12 in #259
• feat(zero-bubble): reorder comm-nodes for batch-p2p by @ChengYao-amd in #257
• refactor(torchtitan): rollback Titan to 99c0cb2(20250907) and stabilize trainer UTs by @Xiaoming-AMD in #262
• feat(turbo): Add turbo RMSNorm patch by @ChengYao-amd in #263
• fix: Disable double DDP construction inside build_model() via runtime patch by @Xiaoming-AMD in #264
• fix(light-megatron): resolve config parsing and adapter compatibility issues by @Xiaoming-AMD in #265
• Support torch_dist async checkpoint for latest Megatron-LM by @limou102 in #267
• New envs: HSA_KERNARG_POOL_SIZE, ENABLE_NUMA_BINDING by @wenxie-amd in #268
• feat: megatron support turbo fp8 grouped gemm by @xiaobochen-amd in #261
• feat(cli): add dataset preparation hooks for train/pretrain workflows by @Xiaoming-AMD in #270
• strided allgather benchmark by @wenxie-amd in #271
• disable cross entropy flags to avoid convergence loss divergence by @clairesonglee in #269
• [Fix] import MXLinear from Primus Turbo by @GeneDer in #272
• add public primus-safe link in readme by @wenxie-amd in #274
• fix readme's typo error by @wenxie-amd in #275
• Add memory projection cli by @yuankaichen-amd in #273
• feat(CLI): Add Runner Library and Test Suite by @Xiaoming-AMD in #276
• feat(runner): add patch execution system with comprehensive test suite by @Xiaoming-AMD in #279
• feat: Support wgrad in MoE overlap by @yuankaichen-amd in #278
• feature(cli): refactor environment configuration with layered design by @Xiaoming-AMD in #280
• refactor: Runner CLI Refactoring and Optimization by @Xiaoming-AMD in #281
• Documentation Reorganization and Structure Improvements by @Xiaoming-AMD in #283
• docs: reorganize backend patch notes + link docs by @Xiaoming-AMD in #284
• feature(cli): cli auto-discover subcommands and add tests by @Xiaoming-AMD in #285
• specify recompute_layer_ids by @llying-001 in #286
• fix(zerobubble): fix zerobubble pp warmup error by @ChengYao-amd in #287
• fix(token_dispatcher): fix deep_ep internode_combine hang when enable sync-free stage 2. by @zhenhuang12 in #289
• fix loading data bug when using vpp by @lhzhang333 in #277
• benchmark: emit GEMM reports as markdown by @Xiaoming-AMD in #288
• support maxtext backend by @llying-001 in #291
• bench: fix dense gemm pre-commit issues by @Xiaoming-AMD in #295
• feat(IRLens): open-source the IRLens tool by @yihuang-amd in #296
• fix errors when patch_moe_overlap is set and distribute Wgrads to overlap with different comms by @yuankaichen-amd in #294
• deprecate decoder_pipeline_manual_split_list by @lhzhang333 in #293
• feat(config): make Megatron workspace path configurable via env var by @poznano-amd in #298

New Contributors

• @GeneDer made their first contribution in #272
• @poznano-amd made their first contribution in #298

Full Changelog: v0.4.0...v0.5.0

v0.4.0

What's Changed

• fix(config): correct flavor to 405B in torchtitan/llama3.1_405B.yaml by @Xiaoming-AMD in #189
• perf(torchtitan/config): enable compile for Llama-3.1 (8B/70B/405B) by @Xiaoming-AMD in #193
• disable dump_pp_data when pp size is one by @lhzhang333 in #191
• remove turbo token by @wenxie-amd in #197
• feat(async-tp) change gemm_rs_overlap api for multi-stream method by @llying-001 in #171
• Support for torchtitan with Primus-Turbo by @clairesonglee in #188
• chore: update default rocm/megatron-lm image to v25.8_py310 by @Xiaoming-AMD in #198
• perf(aiter): add AITER_JIT_DIR env for cached build to speed up re-compilation by @Xiaoming-AMD in #199
• feat: align primus-turbo fp8 linear's args to megatron by @RuibinCheung in #195
• Add wandb_enable config and Torchtitan unit tests by @zitree in #194
• fix: wrapper turbo quant config in megatron extension by @RuibinCheung in #202
• feat(cli): add Python-based primus entrypoint for PATH installation by @Xiaoming-AMD in #200
• feat(zero-bubble): support zero bubble pipeline parallism by @ChengYao-amd in #208
• Primus product matrix by @wenxie-amd in #210
• fix: remove MXQuantConfig from titan and add warning msg by @RuibinCheung in #212
• fix 8B perf regression (v25.9) by @wenxie-amd in #215
• feat(zero-bubble): support GroupGemm wgrad split, add debug_scheduler_table flag by @ChengYao-amd in #213
• add support for grok1 by @JohnQinAMD in #216
• improve torch profiling by @wenxie-amd in #218
• supports: userId for request by @weilei0120 in #214
• support mlflow tracking by @wenxie-amd in #219
• feat: Update Megatron-LM to 8477817(20251011) by @Xiaoming-AMD in #221
• test(megatron): add Qwen2.5-7B and Qwen2.5-72B pretrain cases by @Xiaoming-AMD in #222
• feat(CLI): add unified shell entry scripts for Slurm, container, and direct modes by @Xiaoming-AMD in #209
• Add tensor size print for comm op benchmark by @lorri-rao in #223
• fix(megatron): fix bugs for fitting the newest megatron by @ChengYao-amd in #224
• Docker Release v25.9 by @wenxie-amd in #217
• Add grok2 model support by @wenxie-amd in #227
• Use PRIMUS_xxx env, export all envs for slurm by @wenxie-amd in #229
• feat(deepep): add PrimusTurboDeepEPTokenDispatcher and support syncfree moe stage 0-2 by @zhenhuang12 in #220
• upgrade(torchtitan): sync torchtitan to 5fb7cc2e3bbb9b9dc0ab7af34ed5cc58b5f32021 (2025-10-16) by @Xiaoming-AMD in #228
• chore(docker): update default image to rocm/primus:v25.9_gfx942 by @Xiaoming-AMD in #230
• fix(tests): add missing expecttest dependency for distributed tests by @Xiaoming-AMD in #233
• fix(config): use 1.0e-2 for moe_aux_loss_coeff to ensure correct float parsing by @Xiaoming-AMD in #234

New Contributors

• @clairesonglee made their first contribution in #188
• @zitree made their first contribution in #194
• @lorri-rao made their first contribution in #223

Full Changelog: v0.2.0...v0.4.0