DIRECTORY_CACHE.pop(self.path) never matches a key, so the explicit cache invalidation is dead code

[nkemnitz]

🤖 helped me writing this summary

flush, flush_region and put each invalidate the directory-listing memo with DIRECTORY_CACHE.pop(self.path, None) (:207, :251, :613).

But the memo is keyed on the per-mip subdirectory, not the cache root — compute_data_locations calls no_compression_ext(os.path.join(self.path, list_dir)) (:639-640). self.path is therefore never a key and all three pops are unconditional no-ops.

import tempfile
import numpy as np
from cloudvolume import CloudVolume
import cloudvolume.cacheservice as cs

vol_dir, cache_dir = tempfile.mkdtemp(), tempfile.mkdtemp()
CloudVolume.from_numpy(
    np.zeros((128, 128, 64), dtype=np.uint8) + 7,
    vol_path='file://' + vol_dir, resolution=(4, 4, 40),
    chunk_size=(64, 64, 64), layer_type='image', compress=False,
)
cv = CloudVolume('file://' + vol_dir, cache=cache_dir, compress_cache=False, lru_bytes=0)
cv[0:128, 0:128, 0:64]

print(cv.image.cache.path)          # /tmp/.../file/tmp/...
print(list(cs.DIRECTORY_CACHE))     # ['/tmp/.../file/tmp/.../4_4_40']   <- mip subdir
assert cv.image.cache.path in cs.DIRECTORY_CACHE     # fails

Output:

cs.path              : /tmp/tmpbd8w5exp/file/tmp/tmpu58aglzr
DIRECTORY_CACHE keys : ['/tmp/tmpbd8w5exp/file/tmp/tmpu58aglzr/4_4_40']
cs.path is a key?    : False

Why it matters

It leaves the st_mtime comparison as the only invalidation mechanism, which was presumably not the intent given the pops are there. That's fine for flush() with no preserve, which shutil.rmtrees the directory itself so no_compression_ext hits FileNotFoundError and re-lists regardless. It's less fine for flush(preserve=<bbox>) and flush_region(), which remove individual files and leave the directory in place — there the memo depends entirely on the delete having moved the directory's mtime.

Whether it does is host-dependent, since the kernel advances directory mtime at clock granularity: I measure 4.00 ms steps on one ext4 box (so a delete inside that window is invisible) and effectively per-write granularity on a GCE e2-micro (so no window at all). On a coarse-granularity host a stale present entry means the reader classifies the chunk local, finds nothing, gets content=None, and with fill_missing=True that's silent zeros.

The mtime-window part is timing-dependent and won't reproduce everywhere. The dead pop above is not — it's unconditional.

Suggested fix

Pop the mip directory rather than the cache root. Each of these already knows which directories it touched, so the explicit invalidation becomes reliable and mtime stays as a backstop rather than the sole mechanism.

Verified on 12.14.4.