From 036e70c31f2ba84f1ad17f3311035138defce326 Mon Sep 17 00:00:00 2001 From: Dongji Gao Date: Tue, 28 Jul 2026 11:49:59 -0700 Subject: [PATCH] Qualify SALM Automodel parallelism support Signed-off-by: Dongji Gao --- docs/source/speechlm2/training_and_scaling.rst | 16 ++++++++++++++-- 1 file changed, 14 insertions(+), 2 deletions(-) diff --git a/docs/source/speechlm2/training_and_scaling.rst b/docs/source/speechlm2/training_and_scaling.rst index 4e213319538a..674e465f0f6d 100644 --- a/docs/source/speechlm2/training_and_scaling.rst +++ b/docs/source/speechlm2/training_and_scaling.rst @@ -101,8 +101,10 @@ AutomodelParallelStrategy (SALMAutomodel) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ For ``SALMAutomodel``, the collection provides ``AutomodelParallelStrategy`` which -delegates device mesh creation and parallelism to NeMo Automodel. This strategy -supports FSDP2, TP, PP, CP, EP (MoE), and HSDP. +delegates device mesh creation and parallelism to NeMo Automodel. The strategy can +construct device meshes for FSDP2, TP, PP, CP, EP (MoE), and HSDP, but actual +support depends on the selected LLM architecture, backend, and combination of +parallelism axes. .. code-block:: yaml @@ -165,6 +167,16 @@ optimizations: primitive for routing tokens to experts across GPUs, significantly reducing the communication overhead of Expert Parallelism. +.. important:: + For the native Nemotron-V3 backbone, the validated ``SALMAutomodel`` + parallelism axes are FSDP2, EP, and CP. CP requires packed sequences and + Transformer Engine attention, as described in the Context Parallelism + section below. + + Nemotron-V3 does not provide a TP plan, so set ``tp_size: 1``. HSDP is not + supported in this path, so set ``dp_replicate_size: 1``. PP has not been + validated for this SpeechLM2 configuration, so set ``pp_size: 1``. + Example: training SALMAutomodel with Nemotron Nano V3 on 8 GPUs with EP=8: .. code-block:: bash