# Dynamic VRAM: reused (warm) model produces NaN/black output on VAE decode, fresh load does not

[FR-Mister-T]

After more research issue seem related to dynamic VRAM
#15452 (comment)

Custom Node Testing

• I have tried disabling custom nodes and the issue persists (see how to disable custom nodes if you need help)

Expected Behavior

Expected behavior

All 4 images render correctly, identical in reliability to a fresh model load.

Hello,

Actual Behavior

Summary

With --enable-dynamic-vram active, running multiple prompts back-to-back against the Boogu T2I Turbo official template (a mixed-precision-quantized checkpoint) produces a fully black image on any generation where the diffusion model is reused from an already-resident state, but produces a correct image whenever the model is freshly loaded. VAE precision (--fp16-vae / --bf16-vae / --fp32-vae) does not affect the outcome — the same failure occurs regardless of VAE dtype.

The failure is deterministic and 100% reproducible in a 4-image batch (different seeds, same workflow): images 1 and 3 (fresh model load) came out clean; images 2 and 4 (warm/reused model) came out black, with a RuntimeWarning: invalid value encountered in cast at the final Image.fromarray step, indicating NaN/Inf in the decoded tensor.

Environment

• ComfyUI version: v0.31.0-8-gcbbc9dab (released 2026-08-08)
• comfy-aimdo version: 0.4.13
• comfy-kitchen version: 0.2.28
• comfyui-frontend-package: 1.48.7
• Python: 3.13.15
• PyTorch: 2.12.0+rocm7.14.0
• ROCm version: 7.14
• GPU: AMD Radeon RX 7600 XT (gfx1102), 16368 MB VRAM
• CPU: AMD Ryzen 7 5700X3D (8-core)
• System RAM: 30986 MB
• OS: Linux 7.0.0-29-generic
• Launch flags: --listen 0.0.0.0 --enable-dynamic-vram --fp32-vae
• Model: Boogu T2I Turbo, official ComfyUI template workflow (unmodified), 4-step turbo sampling
• Model uses mixed-precision quantization: Native ops: asym_w4a8_int8, convrot_w4a4, int8_tensorwise

Steps to reproduce

1. Launch ComfyUI with --enable-dynamic-vram (with or without --fp32-vae — both affected).
2. Load the official Boogu T2I Turbo template workflow.
3. Queue 2 or more generations back-to-back (e.g. a batch of 4 with different seeds) without freeing models between them.
4. Observe: the first generation (fresh model load) renders correctly. Every subsequent generation that reuses the already-resident Boogu model (i.e. no Requested to load Boogu line precedes it in the log — it goes straight to Model Boogu prepared for dynamic VRAM loading) renders fully black.

Expected behavior

All 4 images render correctly, identical in reliability to a fresh model load.

Actual behavior

Alternating clean/black output, tracking exactly with fresh-load vs. warm-reuse of the model:

#	Model load type (log evidence)	Result
1	Requested to load Boogu → Model Boogu prepared for dynamic VRAM loading	Clean
2	Model Boogu prepared for dynamic VRAM loading (no preceding Requested to load Boogu)	Black — NaN
3	Requested to load Boogu → Model Boogu prepared for dynamic VRAM loading	Clean
4	Model Boogu prepared for dynamic VRAM loading (no preceding Requested to load Boogu)	Black — NaN

Relevant log excerpt

[20:39:01.695] Requested to load Boogu
[20:39:01.806] Model Boogu prepared for dynamic VRAM loading. 10842MB Staged. 0 patches attached. Force pre-loaded 357 weights: 1664 KB.
[20:39:21.372] 100%|██████████| 4/4 [00:18<00:00, 4.55s/it]
[20:39:21.378] Requested to load AutoencodingEngine
[20:39:21.413] Model AutoencodingEngine prepared for dynamic VRAM loading. 319MB Staged.
[20:39:43.631] Prompt executed in 49.10 seconds        <-- CLEAN, fresh model load

[20:39:43.977] [MultiGPU_Memory_Management] Triggering PromptExecutor cache reset. Reason: cpu_threshold_exceeded
[20:39:44.091] Model Boogu prepared for dynamic VRAM loading. 10842MB Staged.   <-- no "Requested to load Boogu" this time
[20:40:01.988] 100%|██████████| 4/4 [00:17<00:00, 4.45s/it]
[20:40:02.021] Model AutoencodingEngine prepared for dynamic VRAM loading. 319MB Staged.
[20:40:23.610] .../nodes.py:1691: RuntimeWarning: invalid value encountered in cast
  img = Image.fromarray(np.clip(i, 0, 255).astype(np.uint8))
[20:40:23.638] Prompt executed in 39.66 seconds         <-- BLACK, warm/reused model

The same pattern repeats identically for images 3 and 4 later in the same session (see attached full log).

Additional notes / ruled-out causes

• Not a VAE precision issue. Tested with --fp16-vae(default/implicit bf16), --bf16-vae, and --fp32-vae. All three show the same fresh-load-clean / warm-reuse-black pattern. --fp32-vae only appeared to "fix" the issue in earlier single-generation testing because that test happened to be a fresh model load.
• Not caused by low VRAM/OOM on the failing runs. No Ran out of memory / tiled-decode-fallback warning appears immediately before the NaN in this batch — the failure occurs on a plain (non-tiled) decode.
• Disabling dynamic VRAM (--disable-dynamic-vram) is not a viable workaround on this hardware — it removes dynamic VRAM's memory-pressure management entirely, and system RAM usage balloons to ~27.7 GB (out of 30 GB), triggering the Linux OOM killer to kill the ComfyUI process outright (journalctl -k: Out of memory: Killed process ... anon-rss:27738436kB). So this bug can't currently be avoided by turning dynamic VRAM off.
• A [MultiGPU_Memory_Management] Triggering PromptExecutor cache reset event (from the MultiGPU custom node, triggered by CPU RAM >85%) appeared once during testing, immediately before one of the warm-reuse+NaN runs — possibly relevant, possibly coincidental given RAM pressure was already elevated from having the 10GB text encoder + 10GB diffusion model resident. Flagging in case it's a useful lead, but the fresh-load-vs-reuse correlation held even for the warm-reuse case that did not follow a cache-reset event.

Current workaround

Force a full model unload/reload between generations (e.g. Manager's "Unload Models") instead of queuing multiple prompts back-to-back. This is reliable but defeats the purpose of dynamic VRAM's fast warm-model reuse path.

Suspected component

Given the log evidence (staging without a fresh "Requested to load", combined with the model's mixed-precision quantization: asym_w4a8_int8, convrot_w4a4, int8_tensorwise), the bug likely lives in the interaction between the dynamic VRAM warm-reuse path and comfy-aimdo/comfy-kitchen's quantized inference ops — e.g. a dequantization scale, zero-point, or scratch buffer that's computed once on first load and not correctly re-derived/reset on subsequent reuse.

Steps to Reproduce

boogu_batch_repro.log

Debug Logs

comfyui_stdout.log

Other

No response

[asagi4]

I'm seeing similar things happening with Krea2 (XTX 7900 here, otherwise similar env).

In my case, a reliable trigger seems to be if VAE decoding takes enough memory that offloading happens. That's when the model weights seem to be very likely to get corrupted. If the workflow can stay within limits so that no offloading happens, things continue working. I had one case where the VAE weights got corrupted though which was fixed by unloading the VAE and triggering a re-decode from cached outputs without re-running the prompt, so I think it's likely a more generic offload-related memory corruption bug.

[FR-Mister-T]

I tried to switch to a standard FP8 base model instead of INT8 , the issue remain.

Also this PR might be related as well #15436

Correction + root-cause evidence for #15452: NaN originates in the diffusion model under dynamic VRAM, not in VAE decode

Follow-up to my original report. The original diagnosis was wrong and I want to correct it, plus supply instrumented evidence that localizes the fault precisely.

Correction to the original report

The original issue attributed the black images to VAE decode, based on the RuntimeWarning: invalid value encountered in cast at nodes.py:1691. That warning is in SaveImage, not the VAE — it only proves NaN exists by the time the image is saved.

I instrumented the pipeline. The latent is already 100% NaN coming out of the sampler, before the VAE runs at all. The VAE is not involved. --fp32-vae / --bf16-vae / --fp16-vae make no difference; that was a coincidence in my earlier single-image tests.

Also incorrect in the original report: the claimed "fresh load vs. warm reuse" alternating pattern. With more runs, fully-warm runs are frequently clean. The real correlation is stated below.

Environment

• ComfyUI v0.31.0-8-gcbbc9dab (2026-08-08)
• comfy-aimdo 0.4.13, comfy-kitchen 0.2.28
• PyTorch 2.12.0+rocm7.14.0, ROCm 7.14
• AMD Radeon RX 7600 XT (gfx1102), 16368 MB VRAM; 30986 MB system RAM
• Flags: --listen 0.0.0.0 --enable-dynamic-vram --fp32-vae
• Model: Boogu T2I Turbo, official template workflow, 4-step turbo, mixed-precision quantized
  (Native ops: asym_w4a8_int8, convrot_w4a4, int8_tensorwise)

Method

A diagnostic custom node wrapped three things:

1. comfy.sample.sample / sample_custom — NaN check on the returned latent.
2. comfy.sd.VAE.decode — NaN check on input latent and output pixels.
3. comfy.ops.resolve_cast_module_with_vbar — per-module record of whether the
   module was served warm (resident=True) or freshly loaded; whether the cached
   _v_weight pointer still lies inside the module's own VBAR allocation; and a
   sampled NaN check of the cached weight itself, preceded by
   torch.cuda.synchronize() so a hit cannot be an in-flight DMA read.

Then forward hooks on all 993 leaf modules of the diffusion model, recording the
first module emitting NaN from finite inputs — i.e. the origin, independent of
weight-sampling coverage.

Finding 1 — NaN originates in the diffusion model, not the VAE

SAMPLER OUTPUT latent: shape=(1, 16, 96, 171) dtype=torch.float32
  nan=262656 inf=0 min=n/a max=n/a (NO finite values)

262656 = 16×96×171, i.e. 100% of the latent. The VAE then faithfully decodes NaN
into a black image.

Finding 2 — VBAR-resident weights read back as NaN before any compute

On a failing run, sampled cached weights were already NaN when read out of VRAM,
with a synchronized read:

weights_checked=170 weights_with_nan=2
  | NaN weight modules: diffusion_model.time_caption_embed.timestep_embedder.linear_1,
                        diffusion_model.time_caption_embed.timestep_embedder.linear_2

On clean runs, weights_with_nan=0. The cached pointers were always valid
(cached_ptr(inside_own_vbar=2904, OUTSIDE=0) in every run) — so this is not pointer
aliasing into a transient cast buffer; the page contents themselves are corrupt.

Critically, vbar_signature_compare() reported these pages as unchanged, so
resident=True and the reload was skipped. The residency signature is failing to
detect that the backing pages were reclaimed.

Finding 3 — the fault is general, not specific to one layer

Forcing the timestep_embedder modules off the VBAR path (local one-line patch at
comfy/model_patcher.py:1938, if module_mem > 16 * 1024 and "timestep_embedder" not in n:)
dropped the failure rate from 3-in-4 to roughly 1-in-8, and eliminated those
specific NaN weights. But black images continued, now originating elsewhere:

FIRST NaN-EMITTING MODULE: context_refiner.0.feed_forward.linear_1
  [Linear weight=(13568, 3360) on_vbar=True]
  -> *** ORIGIN: inputs finite, output NaN ***

A Linear receiving finite inputs and emitting NaN implies corrupt weights.

Two confirmed corruption sites so far:

Module	Weight	On VBAR	Forward-pass position
time_caption_embed.timestep_embedder.linear_1/2	small	yes	very early
context_refiner.0.feed_forward.linear_1	13568×3360	yes	early (block 0)

Sizes differ by orders of magnitude, so this is not a size-threshold issue. What they
share is early position in the forward pass.

Hypothesis

Early layers are loaded first and evicted first as later layers stream in during the
same forward pass. On a subsequent prompt their VBAR pages have been reclaimed, but the
cached _v_signature still compares equal, so resolve_cast_module_with_vbar() takes
the resident branch (comfy/ops.py:265-267), skips the reload, and computes against
reclaimed memory.

This accounts for every observation:

• The first run after a model load is always clean — the fresh=482 count is exactly
  those early layers being reloaded.
• Intermittency tracks eviction timing rather than seed or prompt.
• Fixing one victim layer just relocates the failure to the next-earliest one.
• timestep_embedder produces a black image ~100% of the time when hit, because it
  feeds every block at every step, so its NaN propagates to the entire latent.

The signature/eviction logic lives in the compiled aimdo_rocm.so, so I can't take this
further from the Python side.

Reproduction

1. Launch with --enable-dynamic-vram on a VRAM-constrained AMD/ROCm setup
   (16 GB card; 10 GB text encoder + 10.8 GB diffusion model cannot both stay resident).
2. Load the official Boogu T2I Turbo template.
3. Queue 8 generations with different seeds.
4. Expect roughly 1-in-8 (up to 3-in-4 unpatched) to be fully black, with a
   100% NaN latent straight out of the sampler.

Notes

• --disable-dynamic-vram is not a usable workaround here: without dynamic VRAM's
  memory-pressure management, system RAM usage reaches ~27.7 GB of 30 GB and the Linux
  OOM killer terminates the process
  (journalctl -k: Out of memory: Killed process ... anon-rss:27738436kB).
• Current workaround is to force a full model unload between generations, which defeats
  the purpose of warm reuse.
• The diagnostic node used to produce all of the above can be supplied if useful.

[asagi4]

vbar_signature_compare in comfy-aimdo seems to be a memoryview comparison. I wonder if there could be a race where a vbar gets freed and then immediately reallocated for something else with the same signature, which would cause them to match against cached pointers even though it no longer belongs to a specific weight. :/

@FR-Mister-T if you point your LLM at the comfy-aimdo source code, maybe it'll find potential sources for that kind of race.

[jnolck]

I'm seeing similar things happening with Krea2 (XTX 7900 here, otherwise similar env).

In my case, a reliable trigger seems to be if VAE decoding takes enough memory that offloading happens. That's when the model weights seem to be very likely to get corrupted. If the workflow can stay within limits so that no offloading happens, things continue working. I had one case where the VAE weights got corrupted though which was fixed by unloading the VAE and triggering a re-decode from cached outputs without re-running the prompt, so I think it's likely a more generic offload-related memory corruption bug.

Tiled decoding helps. At 512 tile size I can make more images without nan or weird output. At 256 it makes the bug trigger 100% of the time.