[feature] Roadmap for DeepSeek-V4 RL training support (DSA/MLA kernels, mixed precision, GPU footprint, and the agentic environment path)

[hyfine]

User problem

I want to run online RL on DeepSeek-V4 (V4-Flash first, V4-Pro later) with Molt, using its agentic Env/ChatAgent reward path. I'm coming to this with two pieces of directly relevant prior experience:

• DeepSeek-V4 Flash/Pro SFT on NVIDIA AutoModel - got it working, but it needed modifications. AutoModel was not turnkey for V4: I had to work around mixed-precision (fp8) issues and the chat-template gap (V4 ships no HF tokenizer.chat_template - formatting is done by a Python encoder, encoding/encoding_dsv4.py::encode_messages, not Jinja). Since Molt's training backend is AutoModel + FSDP2, I expect the same two issues to resurface on the RL path unless they're handled explicitly.
• DeepSeek-V4 GRPO on verl - ran into a long tail of environment pitfalls, most driven by our cluster constraints: driver 570.124.06 with a CUDA 12.8 ceiling. The single hardest wall was structural: DeepSeek-V4 is only recognized by vLLM's ModelRegistry from ≥0.20.0, but every PyPI vLLM wheel ≥0.20.0 links CUDA 13 and fails CUDA init against a 570.x driver - I only got past it by source-building vLLM 0.22.1 against CUDA 12.2. On top of that: transformers ≥5.8.0 needed for deepseek_v4, TransformerEngine requiring CUDA <13.0, FlashMLA nv_dev needing nvcc ≥12.9, Megatron-core API drift, and a series of multi-node-only failures (Ray GCS registration timeouts, PYTHONPATH not reaching non-head actors, hardcoded Ray port collisions on a shared cluster, etc.) that were invisible to single-node dry-runs.

Given all that, my core ask is not just "does Molt plan to support V4 RL" - it's whether Molt can offer a more user-friendly, batteries-included recipe than what I had to assemble by hand on verl. Concretely, a V4 RL recipe that ships:

1. A one-command environment setup script that resolves the known version/toolchain triangle end-to-end (transformers / vLLM / TransformerEngine / FlashMLA / cuDNN-frontend / Megatron-core), with a documented, driver-aware path for clusters stuck below CUDA 13 (e.g. a source-build option for the rollout engine) - so users don't rediscover the 570.x-driver wall the hard way.
2. A ready training config for V4-Flash (and Pro) that already encodes the hard architecture constraints - TP=1, PP×EP×CP mesh, MTP-on-last-stage, fp8/mixed-precision settings, and the chat-template injection - so it works without per-user patching.
3. Multi-node launch scripts for the common cluster shapes - a SLURM sbatch script and a bare-metal multi-node Ray bring-up script — that handle the operational details that only bite at scale (network/IB interface detection, staggered Ray head/worker startup with a registration gate, non-default port selection on shared clusters, PYTHONPATH/env propagation to remote actors).

Desired outcome

A stated plan (and ideally a validated reference recipe) covering:

• Model path decision
• Precision story
• A user-friendly V4-Flash RL recipe delivering the three items above
• A supported vLLM build matrix
• Agentic-path confirmation

Affected area

area:llm

Extra context

Affected areas in Molt: the AutoModel model backend (native V4 class / DSA-MLA), molt/trainer/fsdp/ (PP/EP/CP with tp_size=1, MTP-last-stage layout, optimizer offload), molt/trainer/vllm/ (engine build + DeepseekV4ForCausalLM recognition + CUDA matrix; Molt currently pins vllm==0.24.0 — recognition/CUDA matrix unknown to me), MoE routing stability, and the agent/chat-template path in molt/agents/

[hijkzzz]

NVIDIA Automodel already supports DeepSeek V4, but we are currently refactoring the context parallel API. It is almost finished. Once that is done, we will add this support and should be able to fully support DeepSeek V4 with context parallel. Otherwise, some hacky code would be required, which I feel is not very elegant.

[hijkzzz]

The cuda13 docker image has been fixed.
Deepseek V4 + FP8 is working on progress (see NVIDIA-NeMo/Automodel#3252).
You could try to use th Automodel deepseekv4 branch Now