feat(multimodal): migrate Qwen3.5-VL to HybridModel

[BestJuly]

What

Migrate the examples/multimodal_dev Qwen3.5-VL language decoder from
GPTModel to HybridModel, ahead of the deprecation of the GPTModel code path.

Why

The Qwen3.5-VL decoder is a hybrid GatedDeltaNet / full-attention stack. On the
GPT path that structure was expressed implicitly through
--experimental-attention-variant gated_delta_net plus
--linear-attention-freq 4, with the attention/MLP ordering baked into
TransformerLayer. HybridModel expresses it explicitly through
--hybrid-layer-pattern, which is where this model family belongs long term.

Changes

Model wiring (examples/multimodal_dev)

• MultimodalModel / Qwen35VLModel now take hybrid_stack_spec and
  hybrid_layer_pattern instead of a transformer_layer_spec and an
  mtp_block_spec. get_qwen35_vl_language_spec is removed.
• Each historical Qwen block becomes two hybrid layers: a GatedDeltaNet (G)
  or full-attention (*) layer followed by a dense (-) or MoE (E) MLP
  layer. Three GDN blocks then one attention block reproduces the
  --linear-attention-freq 4 cadence exactly.
• MTP depths replicate the last block, matching get_gpt_mtp_block_spec, which
  derived the MTP spec from the final decoder layer.
• Variant configs are converted from block counts to hybrid layer counts,
  because HybridModel counts one layer per pattern symbol.

mRoPE support in HybridModel (megatron/core/models/hybrid/hybrid_model.py)

• Build a MultimodalRotaryEmbedding when position_embedding_type == 'mrope'
  and compute sectioned (optionally raw, fused-kernel) freqs in forward,
  mirroring GPTModel line for line, including the
  apply_rope_fusion / rotary_interleaved / fused_single_qkv_rope /
  inference conditions and the flash-decode NotImplementedError.
• Widen the HybridModelConfig.position_embedding_type literal accordingly.

GatedDeltaNet (megatron/core/ssm/gated_delta_net.py)

• Import fla.ops.cp.build_cp_context separately from the rest of FLA so an FLA
  build without the CP extension still enables GatedDeltaNet, and raise a
  descriptive error only when chunkwise CP is actually requested.

Launcher (scripts/run_qwen35_vl.sh)

• Derive the pattern from NUM_LAYERS, NUM_EXPERTS and the new
  MTP_NUM_LAYERS knob; pass the MTP arguments only when MTP is enabled.
• NUM_LAYERS need not be a multiple of 4: trailing blocks stay GatedDeltaNet,
  which is what --linear-attention-freq 4 produced, so shallow proxy runs keep
  working.

Tests

• examples/multimodal_dev/tests/test_hybrid_migration.py — factory wiring plus
  structural equivalence between the hybrid pattern and the former GPT layout.
  The key check cross-validates validate_segment_layers (hybrid) against
  get_linear_attention_pattern (GPT) so a semantic drift on either side fails.
• tests/unit_tests/models/test_hybrid_model.py — HybridModel constructor with
  position_embedding_type='mrope'.
• examples/multimodal_dev/tests/test_cp_thd_correctness.py — ported to the
  hybrid stack spec.

Checkpoint compatibility

The decoder checkpoint layout changes: one GPT decoder layer is split across two
hybrid layer indices, the final-norm key changes, and hybrid MTP uses a nested
HybridStack. An existing GPTModel-format Qwen3.5-VL checkpoint is therefore not
directly loadable. tools/checkpoint/gpt_hybrid_conversion.py does not cover
this case either — it explicitly rejects GDN (G) and MTP checkpoints. This is
documented in examples/multimodal_dev/README.md.

Scope of risk

The megatron/core changes are additive: the mRoPE branches only run when
position_embedding_type == 'mrope', which no existing HybridModel
configuration uses, and the FLA import change only alters behavior on builds
that lack fla.ops.cp. Everything else is confined to examples/multimodal_dev.

[copy-pr-bot]

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.