Summary
Compiler::build() handles concurrent builds of the same kernel by compiling into a private temp directory and atomically renaming it into the cache; a losing process deletes its temp directory and re-reads the winner's (csrc/jit/compiler.hpp:156-158):
const auto runtime = kernel_runtime_cache->get(dir_path);
EP_HOST_ASSERT(runtime != nullptr);
This assumes a failed rename implies the winner's directory is visible to the loser. On a local filesystem that is guaranteed; on NFS it is not, and that is the default configuration, since the cache falls back to $HOME/.deep_ep (compiler.hpp:52) and home directories are commonly NFS on multi-node clusters. The losing rank dies on the assert and kills the job. Observed on 2.0.0 (EPv2); code unchanged on current main (60d4403).
Mechanism
Cold start, all ranks race build() for the same kernel signature:
- Every rank misses at
compiler.hpp:116. that stat primes each node's NFS client negative lookup cache for dir_path ("does not exist"), valid until the parent directory's attribute cache expires (up to ~60 s on default mounts).
- All ranks compile in parallel into private
<cache>/tmp/<uuid> dirs; the first finisher's rename lands atomically at the server.
- A loser on another node gets an authoritative rename failure, deletes its temp dir, and re-stats
dir_path. Answered "does not exist" from its still-valid negative cache entry, without a server round-trip. check_validity → false, get() → nullptr, assert at compiler.hpp:158.
The existing fsync/atomic-rename hardening is writer-side (durability and ordering at the server); this failure is reader-side, in the loser's own client cache, which writer-side fsync cannot invalidate. A failed rename() does not invalidate the caller's cached lookup of the target either.
Reproduced on the first attempt with 16 ranks across two 8-GPU H200 nodes and a cold NFS cache: all 8 ranks of the losing node crashed at the assert while JIT-building the dispatch/combine kernels (whole-node failure as the stale cache belongs to the node's NFS client), and the winner node's ranks hung in the next collective until the job was killed.
GB200/GB300-class systems (4 GPUs per node) are more exposed: every EP ≥ 8 job spans multiple NFS clients, so a shared cache races from the smallest multi-node shape (a GB300 deployment we checked shares an NFSv4.2 filesystem with default caching options). The interconnect is irrelevant — the race is in the host-side compile path, identical on aarch64.
Proposed fix: #684
The loser does not need the winner's directory: it holds freshly compiled, equivalent artifacts (the cache key hashes kernel name, source, flags, and compiler signature). Load the runtime from its own temp directory, cache it in-memory under the canonical key, then delete the temp dir. This is safe because KernelRuntime's constructor consumes the files eagerly (cuobjdump + cudaLibraryLoadFromFile/cuModuleLoad) and keeps only GPU handles; warm starts still read the shared cache as before, and other rename failures degrade gracefully instead of asserting.
Validated on the reproducing setup: the crashing scenario passes 16/16 with the fix, with the losing branch exercised 73 times across 80 rank×kernel builds (verified via EP_JIT_DEBUG=1 load paths); a warm-start rerun serves all 80 loads from the canonical cache directories.
Notes
Summary
Compiler::build()handles concurrent builds of the same kernel by compiling into a private temp directory and atomically renaming it into the cache; a losing process deletes its temp directory and re-reads the winner's (csrc/jit/compiler.hpp:156-158):This assumes a failed rename implies the winner's directory is visible to the loser. On a local filesystem that is guaranteed; on NFS it is not, and that is the default configuration, since the cache falls back to
$HOME/.deep_ep(compiler.hpp:52) and home directories are commonly NFS on multi-node clusters. The losing rank dies on the assert and kills the job. Observed on 2.0.0 (EPv2); code unchanged on currentmain(60d4403).Mechanism
Cold start, all ranks race
build()for the same kernel signature:compiler.hpp:116. that stat primes each node's NFS client negative lookup cache fordir_path("does not exist"), valid until the parent directory's attribute cache expires (up to ~60 s on default mounts).<cache>/tmp/<uuid>dirs; the first finisher's rename lands atomically at the server.dir_path. Answered "does not exist" from its still-valid negative cache entry, without a server round-trip.check_validity→ false,get()→nullptr, assert atcompiler.hpp:158.The existing fsync/atomic-rename hardening is writer-side (durability and ordering at the server); this failure is reader-side, in the loser's own client cache, which writer-side fsync cannot invalidate. A failed
rename()does not invalidate the caller's cached lookup of the target either.Reproduced on the first attempt with 16 ranks across two 8-GPU H200 nodes and a cold NFS cache: all 8 ranks of the losing node crashed at the assert while JIT-building the dispatch/combine kernels (whole-node failure as the stale cache belongs to the node's NFS client), and the winner node's ranks hung in the next collective until the job was killed.
GB200/GB300-class systems (4 GPUs per node) are more exposed: every EP ≥ 8 job spans multiple NFS clients, so a shared cache races from the smallest multi-node shape (a GB300 deployment we checked shares an NFSv4.2 filesystem with default caching options). The interconnect is irrelevant — the race is in the host-side compile path, identical on aarch64.
Proposed fix: #684
The loser does not need the winner's directory: it holds freshly compiled, equivalent artifacts (the cache key hashes kernel name, source, flags, and compiler signature). Load the runtime from its own temp directory, cache it in-memory under the canonical key, then delete the temp dir. This is safe because
KernelRuntime's constructor consumes the files eagerly (cuobjdump+cudaLibraryLoadFromFile/cuModuleLoad) and keeps only GPU handles; warm starts still read the shared cache as before, and other rename failures degrade gracefully instead of asserting.Validated on the reproducing setup: the crashing scenario passes 16/16 with the fix, with the losing branch exercised 73 times across 80 rank×kernel builds (verified via
EP_JIT_DEBUG=1load paths); a warm-start rerun serves all 80 loads from the canonical cache directories.Notes
EP_JIT_CACHE_DIRat a node-local path (e.g. under/tmp).flock()(cf. [Bug] JIT cache lacks cross-process synchronization, can cause failures under multi-process parallelism DeepGEMM#301) would reduce the N-way duplicate compilation but not fix this crash: acquiring a lock does not invalidate the client's cached lookup of an unrelated path, andflockis not cross-node onnolockNFSv3 mounts.EP_JIT_DUMP_*set, the losing rank's dumps are deleted with its temp dir under the fix; the winner's remain in the cache.