NaN/Inf Occurs When Running MiniMax H3 with FP16 Precision (Tesla V100 16GB)

[aaalll12322]

Custom Node Testing

• I have tried disabling custom nodes and the issue persists (see how to disable custom nodes if you need help)

Your question

I am using a Tesla V100 16GB graphics card (SM70 architecture) on vanilla ComfyUI with all custom nodes disabled. When I set the calculation precision of MiniMax H3 to FP16 via the ModelComputeDtype node, NaN and Inf values appear in tensors during inference, leading to failed generation.
Switching the precision to FP32 enables normal operation, yet the inference speed of FP32 is only 1/4 of the expected FP16 speed, resulting in severe performance degradation.
I would like to ask if there is a fix available for this FP16 numerical overflow issue.

Logs

Other

No response

[johnrutten]

pls show workflow ?

[aaalll12322]

请展示工作流程？

The workflow is the official ComfyUI Image-to-Video template from the template library. The only modification is that I added a ModelComputeDtype node right after the UNETLoader and set its dtype to fp16. The model being used is minimax_h3_fl2va_pruned_fp8_scaled.safetensors. All other nodes and parameters remain exactly as they are in the original official template.

video_minimax_h3_i2v.json

[dg1kjd]

Hi @aaalll12322 — we've been bringing H3 up on 8×V100-SXM2-32GB (sm_70) and hit NaN from two independent causes. Since this was closed without a recorded fix, here's what we found. The first one explains your report directly.

1. Why fp16 compute can't work for H3 — this is your issue

This isn't something you can configure around. H3's DiT carries "massive activations": we instrumented the residual stream and it reaches ~1e6–1e7 by mid-stack. fp16 tops out at 65504, so forcing fp16 compute overflows almost immediately. That's exactly why fp32 works for you.

On our box ComfyUI selects model weight dtype torch.bfloat16, manual cast: torch.float32. The residual must be fp32 — or bf16, which shares fp32's exponent range. On Ampere+ bf16 gives you that range at tensor-core speed; on Volta there's no bf16 hardware, so fp32 is the only correct option. So: no fix for fp16 compute. It's structural to the model, not specific to Volta.

2. A second, independent NaN source — worth knowing even in fp32

We separately hit NaN while running fp32, which took much longer to pin down. Root cause: comfy-aimdo's dynamic-VRAM vbar weight paging issues vbar_unpin (an immediate host-side free) while the consuming GEMM/cast kernels are still queued on the CUDA stream, with no stream or event sync. The op reads freed VRAM → straight to NaN.

Signature:

• NaN with isinf == False — garbage bit patterns, not overflow
• nondeterministic at a fixed seed — this is the giveaway
• worse with longer sequences and more steps (more VRAM pressure → more paging)
• an instrumented probe caught the first NaN at final_layer.audio_out with all inputs finite

Fix: launch with --disable-dynamic-vram (boot log should read "Dynamic vram disabled" → NORMAL_VRAM). A config that reliably NaN'd by step 7–8 then ran clean 20- and 24-step renders at 1344×768/124f. Costs roughly 30% step time from losing async weight paging.

On a 16 GB V100 you'll be paging hard, so this is fairly likely to bite you once fp32 is working.

3. On "1/4 of the expected fp16 speed"

That's expected on Volta: fp32 is ~15.7 TFLOPS with no tensor-core path, versus ~125 TFLOPS for fp16-input/fp32-accumulate. The ~8× ratio is roughly the gap you're measuring.

One open question we haven't settled: whether attention specifically could run fp16-in/fp32-accumulate while the residual stays fp32 — q/k are RMSNormed and bounded, but v isn't and would need scaling. Untested on our side; flagging in case it's useful to you.

[aaalll12322]

Hi @aaalll12322 — we've been bringing H3 up on 8×V100-SXM2-32GB (sm_70) and hit NaN from two independent causes. Since this was closed without a recorded fix, here's what we found. The first one explains your report directly.

1. Why fp16 compute can't work for H3 — this is your issue

This isn't something you can configure around. H3's DiT carries "massive activations": we instrumented the residual stream and it reaches ~1e6–1e7 by mid-stack. fp16 tops out at 65504, so forcing fp16 compute overflows almost immediately. That's exactly why fp32 works for you.

On our box ComfyUI selects . The residual must be fp32 — or bf16, which shares fp32's exponent range. On Ampere+ bf16 gives you that range at tensor-core speed; on Volta there's no bf16 hardware, so fp32 is the only correct option. So: no fix for fp16 compute. It's structural to the model, not specific to Volta.model weight dtype torch.bfloat16, manual cast: torch.float32

2. A second, independent NaN source — worth knowing even in fp32

We separately hit NaN while running fp32, which took much longer to pin down. Root cause: comfy-aimdo's dynamic-VRAM weight paging issues (an immediate host-side free) while the consuming GEMM/cast kernels are still queued on the CUDA stream, with no stream or event sync. The op reads freed VRAM → straight to NaN.vbar``vbar_unpin

Signature:

• NaN with — garbage bit patterns, not overflowisinf == False
• nondeterministic at a fixed seed — this is the giveaway
• worse with longer sequences and more steps (more VRAM pressure → more paging)
• an instrumented probe caught the first NaN at with all inputs finitefinal_layer.audio_out

Fix: launch with (boot log should read "Dynamic vram disabled" → ). A config that reliably NaN'd by step 7–8 then ran clean 20- and 24-step renders at 1344×768/124f. Costs roughly 30% step time from losing async weight paging.--disable-dynamic-vram``NORMAL_VRAM

On a 16 GB V100 you'll be paging hard, so this is fairly likely to bite you once fp32 is working.

3. On "1/4 of the expected fp16 speed"

That's expected on Volta: fp32 is ~15.7 TFLOPS with no tensor-core path, versus ~125 TFLOPS for fp16-input/fp32-accumulate. The ~8× ratio is roughly the gap you're measuring.

One open question we haven't settled: whether attention specifically could run fp16-in/fp32-accumulate while the residual stays fp32 — q/k are RMSNormed and bounded, but v isn't and would need scaling. Untested on our side; flagging in case it's useful to you.

Thanks for the detailed analysis. The fp16 NaN issue is resolved by this plugin:
https://github.com/aaalll12322/ComfyUI-MiniMaxH3-FP16Safe
Tested on V100, runs stably without NaN.

[dg1kjd]

@aaalll12322 — apologies. You were right and I was wrong. Your plugin was published three days before my comment, it does exactly what I claimed couldn't be done, and my own closing paragraph admitted the key question was untested. I should have checked before asserting.

We've since been running your plugin on an 8×V100-SXM2-32GB box while working on our own sequence-parallel setup for multi-GPU machines (I'll post separately about that when it's ready). Along the way we measured four things about your code that you may find useful.

1. Your attention prescale can be /16 instead of /256 — measurably more accurate, same safety

RMSNorm is scale-homogeneous only as eps → 0:

rms_norm(q/s) = q / sqrt(ms_q + eps·s²)

H3 sets qk_norm_eps = 1e-5, so at s=256 the non-cancelling term is eps·s² = 0.655. That is not small next to the live magnitudes — we probed ms(q) at block 0 of 50 and get median ≈ 9–10, min ≈ 2.2.

Attention output vs an unprescaled fp32 reference on identical activations (1344×768/124f, so no sampling-chaos confound):

prescale	cos	max rel err
/256 (current)	0.99973989	7.2e-2
/64	0.99999893	5.0e-3
/32	0.99999994	1.25e-3
/16	1.00000000	3.1e-4
/8	1.00000000	7.7e-5

The error follows exactly 1/s² with no plateau, so there's no sweet spot — smaller is monotonically better and the only limit is overflow.

The binding constraint is out_proj's output, the largest fp16 tensor in the attention path. We measure amax 1.43e5 at 1344×768 (you cite ~39k at 480p). At s=16 that still leaves 7.3× headroom on our worst case, ~27× on yours, with your isfinite fuse behind it. Same 1/s² law on both int8_convrot and fp8_scaled checkpoints.

Your MLP factors are a different story — /16 gate + /8 fc2 = /128 total is correct for your workload. Only the attention prescale is oversized. One caveat if you ever scale up: we measure mlp output peaking at 4.5e6 at 1344×768 against your ~501k, which would leave /128 only ~1.9× of headroom.

2. Four device syncs per block — 7.8% measured

_fp16_safe() calls h.abs().max().item() twice per DiTBlock (attention input, MLP input), and the two torch.isfinite(...).all() fuses are evaluated in Python ifs. Each forces a GPU→CPU sync: 200 per forward, ~8,000 per 20-step render at cfg>1.

Removing them on your plugin (fp8, 512²/L=39, 8 steps): 5.64 → 5.23 s/it, with output metrics identical (px_std 69.5 vs 69.5, audio peak 0.824 vs 0.824).

The magnitude probes are unnecessary given your fixed scales — we measure the inputs at ≤2.3e2 (attention) and ≤9.3e1 (MLP), far inside range. The fuses are worth keeping, but deferred: accumulate a flag on-GPU and check once per forward instead of 100 times. This should pay more on your 16 GB card than on ours, since a sync also stalls the weight prefetch — you already cite sync-amplification in low-VRAM offload as something you fought in v6.3.

3. Why fp8 beats int8 for you — the cause behind your README's ~9 s/step

F.linear dispatches on the weight, not the activation dtype.

• int8_convrot → comfy-kitchen registers an aten.linear override for TensorWiseINT8Layout, so it goes to torch._int_mm (Volta DP4A, measured ~43 TFLOP/s) no matter what dtype you feed it. Casting activations to fp16 cannot reach the tensor cores.
• fp8_scaled → comfy.model_management.supports_fp8_compute() returns False for props.major < 8, so pick_operations() never selects fp8_ops. The weight is dequantised to the compute dtype and run dense — no fast path, so the compute dtype is the whole story, and your fp16 cast does reach the tensor cores (~81 TFLOP/s).

Per block, four real linears at S=4725: fp32 dense 255.7 ms → fp16 via fp8 dequant 49.0 ms; int8 DP4A sits at 83.0 ms. So fp8 is the right recommendation on sm_70, and on int8 checkpoints the fp16 GEMM path isn't where the win is.

4. A bug: the forced compute dtype survives removing your node

set_model_compute_dtype(torch.float16) mutates the ModelPatcher, and ComfyUI caches the UNETLoader output across runs. Delete your node from the workflow, re-run, and the model is still forced to fp16 compute — now without your scaling compensation, which is exactly this issue's original NaN/black-frame condition.

It bit us: a run with your node removed died with avcodec_send_frame() returned 22; a clean-server A/B confirmed the cause (fresh process → fp32 compute → healthy). Your v6.5.0 notes say instance-level patching fixed ghost patches — it fixed the forward patches, but not the dtype mutation. Restoring the previous compute dtype on cleanup should cover it.

One thing we checked that turned out fine

We'd seen fp16 GEMMs rail H3's audio in our own runner and suspected your global scales might do the same — the packed sequence mixes video and audio tokens differing by up to 1375× in RMS at block 0. We tested it: your audio is clean. Exact powers of two don't have that failure mode; a fitted or per-tensor exposure would. Worth knowing if you ever change the scaling approach.

Thanks for the plugin, and again, sorry for the noise.

[aaalll12322]

Thanks @dg1kjd — apology accepted, and more importantly, thanks for the follow-up measurements. Closing the loop on your four points:

1. /16 prescale — merged (your PR, your commit and authorship). Independent verification on a real model, 52 layers: mean rel err 1.35e-2 → 5.6e-5 (~240× ≈ (256/16)², strictly monotonic), worst-case out_proj fp16 output 1931 (33.9× headroom here). Details in DEVELOPMENT.md §3.

2. Sync cost — agree, planned. 7.8% on your box, worse on 16 GB where a sync also stalls weight prefetch. Next version moves magnitude checks to once-per-forward (deferred fuse), keeping the isfinite backstop.

3. fp8 vs int8 mechanism — thanks for confirming. Matches our README: fp8_scaled dequant→fp16 hits the tensor cores (~81 TFLOP/s); int8_convrot stays on DP4A (~43 TFLOP/s).

4. Compute-dtype leak — fixed (v6.6.0 dtype side + v6.7.0 forward side). v6.6.0 clones the ModelPatcher and patches the clone, so the cached UNETLoader output no longer retains the fp16 compute dtype. We then found and fixed (v6.7.0) the second half: ModelPatcher.clone() still shares the model instance, so forward patches could leak into the cached object (deleting the node kept the plugin forward active until reload). v6.7.0 structure-clones the module tree at patch time — params shared, module instances isolated — so cached objects always keep their native forward. Both sides verified with dedicated scripts in the repo.

Also noted: mlp peak 4.5e6 at 1344×768 (vs our ~501k) leaves /128 at ~1.9× — flagged in DEVELOPMENT.md if we ever scale up. And glad the audio check came out clean; the 2^k scales were chosen partly for exactly that reason.

Re: "closed without a recorded fix" — fair hit. For the record, the one-liner is: fp16 compute works on H3 if you (a) keep the residual fp16-safe with exact 2^k prescaling through the linear chain, (b) keep a fused isfinite backstop, and (c) isolate every patch on a ModelPatcher clone with a structure-cloned module tree so ComfyUI's cache stays clean. Full writeup in the plugin's DEVELOPMENT.md. https://github.com/aaalll12322/ComfyUI-MiniMaxH3-FP16Safe

Best,
aaalll12322

[dg1kjd]

Following up on the sequence-parallel work I mentioned — it's published now:
https://github.com/dg1kjd/comfyui-v100-sxm2-minimax-h3

To be clear about scope, since this thread is about a different problem: this is only useful if you have several GPUs. It splits one generation across them (Ulysses sequence parallelism) rather than fixing anything numerical. On 8× V100 the canonical 1344×768 / 124-frame clip goes from ~19 minutes to about 7.

If you're on a single V100 — which is this issue's case — @aaalll12322's ComfyUI-MiniMaxH3-FP16Safe is what you want, not this. Different problem, and it's the one that solves yours. The two are complementary rather than alternatives.

GPL-3.0, drop-in (no ComfyUI files modified), example workflows included. Tuned for a DGX-1 style NVLink topology; correct but likely not optimal elsewhere.