Skip to content
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 14 additions & 2 deletions docs/source/speechlm2/training_and_scaling.rst
Original file line number Diff line number Diff line change
Expand Up @@ -101,8 +101,10 @@ AutomodelParallelStrategy (SALMAutomodel)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

For ``SALMAutomodel``, the collection provides ``AutomodelParallelStrategy`` which
delegates device mesh creation and parallelism to NeMo Automodel. This strategy
supports FSDP2, TP, PP, CP, EP (MoE), and HSDP.
delegates device mesh creation and parallelism to NeMo Automodel. The strategy can
construct device meshes for FSDP2, TP, PP, CP, EP (MoE), and HSDP, but actual
support depends on the selected LLM architecture, backend, and combination of
parallelism axes.

.. code-block:: yaml

Expand Down Expand Up @@ -165,6 +167,16 @@ optimizations:
primitive for routing tokens to experts across GPUs, significantly reducing the
communication overhead of Expert Parallelism.

.. important::
For the native Nemotron-V3 backbone, the validated ``SALMAutomodel``
parallelism axes are FSDP2, EP, and CP. CP requires packed sequences and
Transformer Engine attention, as described in the Context Parallelism
section below.

Nemotron-V3 does not provide a TP plan, so set ``tp_size: 1``. HSDP is not
supported in this path, so set ``dp_replicate_size: 1``. PP has not been
validated for this SpeechLM2 configuration, so set ``pp_size: 1``.

Example: training SALMAutomodel with Nemotron Nano V3 on 8 GPUs with EP=8:

.. code-block:: bash
Expand Down
Loading