[KERNELS] Fix Blackwell NVFP4 x dense matmul tile selection

[amyyang-openai]

Summary

• extend the persistent Blackwell TMEM-safe block_n <= 128 rule to NVFP4 activation × dense BF16/FP16 matmuls with 16-element activation microblocks
• keep the existing MXFP predicate unchanged, so its independent block_k and eight-warp selection remain untouched
• preserve explicit planner constraints, Hopper behavior, and the GB300 large-routed NVFP4 × NVFP4 specialization from Fix large ragged NVFP4 tile selection on GB300 #11080

Root cause

For routed M=256, N=256, K=128 NVFP4 activation × dense BF16/FP16 weight matmuls with Blackwell-swizzled activation scales, upstream main selects BLOCK_M=128, BLOCK_N=256, BLOCK_K=256, num_warps=8. The persistent dot requires 640 tensor-memory columns, exceeding Blackwell's 512-column limit:

triton.runtime.errors.OutOfResources:
out of resource: tensor memory, Required: 640, Hardware limit: 512

The existing dense-RHS safeguard only recognizes 32-element MXFP activation microblocks; NVFP4 uses 16-element microblocks. Add a separate narrowly scoped predicate for Blackwell FP4 activations with 16-element microscaling, no RHS/output microscaling, and a BF16/FP16 RHS. Use that predicate only for the persistent block_n safety clamp. It deliberately does not participate in the existing MXFP block_k or num_warps decisions, yielding the safe 128 × 128 × 256, four-warp NVFP4 configuration.

Validation

Base: df7fa7bd086c01e2fdc91bd39676d02511b783f3 (upstream main, 2026-08-04).

Before the fix, reproduced the exact Required: 640, Hardware limit: 512 exception for both BF16 and FP16 on both actual GPU generations.

Hardware	Cluster/quota	RHS	Selected tile	Warps	Maximum relative error	RMS relative error
NVIDIA GB200, SM 10.0	xenon-02-27 / rl-scaling	BF16	128 × 128 × 256	4	0.0	0.0
NVIDIA GB200, SM 10.0	xenon-02-27 / rl-scaling	FP16	128 × 128 × 256	4	0.00714284	2.79104e-5
NVIDIA GB300, SM 10.3	f60 / mlexec	BF16	128 × 128 × 256	4	0.0	0.0
NVIDIA GB300, SM 10.3	f60 / mlexec	FP16	128 × 128 × 256	4	0.00714284	2.79104e-5

On each GPU:

pytest -s --tb=short --device cuda \
  tests/test_matmul_details/test_opt_flags_nvidia.py \
  tests/test_matmul_details/test_opt_flags_split_k.py

11 passed, 4 skipped

CPU/interpreter coverage: 7 passed, 8 skipped. One focused, parameterized routed NVFP4 × BF16/FP16 Blackwell execution test covers the failure end to end. Explicit planner constraints, existing MXFP behavior, Hopper behavior, and the large routed NVFP4 × NVFP4 specialization remain unchanged.