Prewarm the resolver cache to cut serial resolve round-trips - #77
Prewarm the resolver cache to cut serial resolve round-trips#77timkeller wants to merge 4 commits into
Conversation
|
I'd like to work on this bounty. |
There was a problem hiding this comment.
Pull request overview
This PR introduces an opt-out prewarm pass that walks a USD layer graph breadth-first and batch-resolves AYON URIs up front, seeding the process-wide resolver cache to reduce serial server round-trips during USD composition (especially impactful over WAN links).
Changes:
- Add a prewarm BFS pass (
prefetch/prewarm.{h,cpp}) and invoke it once per distinct root asset in_CreateDefaultContextForAsset, controlled by env vars. - Expose the process-wide cache via
GetResolverGlobalCache()so separate translation units (and host entry points) can seed the same cache instance. - Add
ResolverContextCache::batchWarm()to batch-resolve and insert results into the cache.
Reviewed changes
Copilot reviewed 9 out of 10 changed files in this pull request and generated 7 comments.
Show a summary per file
| File | Description |
|---|---|
| src/AyonUsdResolver/resolverContext.h | Declares a global accessor for the process-wide resolver cache. |
| src/AyonUsdResolver/resolverContext.cpp | Defines the new global cache accessor bridging the existing internal cache. |
| src/AyonUsdResolver/resolver.cpp | Triggers prewarm once per root asset and adds env-var gating. |
| src/AyonUsdResolver/prefetch/prewarm.h | Declares the prewarm API and a C entry point for host-side invocation. |
| src/AyonUsdResolver/prefetch/prewarm.cpp | Implements BFS layer-walk + batched warming and enqueues next-frontier layers. |
| src/AyonUsdResolver/CMakeLists.txt | Adds prefetch/prewarm.cpp to the core resolver library sources. |
| src/AyonUsdResolver/cache/resolverContextCache.h | Declares batchWarm() on the resolver cache. |
| src/AyonUsdResolver/cache/resolverContextCache.cpp | Implements batchWarm() using batchResolvePathSerial() and inserts results. |
| .gitignore | Ignores local build/install artifacts and log files. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| #include <cstdlib> | ||
| #include <mutex> | ||
| #include <unordered_set> |
| // Accessor for the single process-wide resolver cache shared by every | ||
| // AyonUsdResolverContext. Exposed so the prewarm pass (a separate translation | ||
| // unit) and the host-side C entry point can seed the same cache instance. | ||
| std::shared_ptr<ResolverContextCache> GetResolverGlobalCache(); |
| * @param uriPaths The AYON URIs to resolve. May be reordered/deduplicated. | ||
| * @return Map of URI -> resolved path for the entries that were resolved. | ||
| */ | ||
| std::unordered_map<std::string, std::string> batchWarm(std::vector<std::string> &uriPaths); |
| * @brief Resolve many AYON URIs in a single batched (parallel) request and seed the cache. | ||
| * | ||
| * Unlike getAsset(), which resolves one URI per server round-trip, this collapses a whole | ||
| * set of URIs into one batched call via AyonApi::batchResolvePath and inserts every result. | ||
| * Used by the prewarm pass to avoid the serial resolve storm during stage composition. |
| * round-trips). Each BFS frontier of AYON URIs is resolved in a single batched, | ||
| * parallel request and inserted into the process-wide resolver cache. By the time | ||
| * USD composes the stage and calls _Resolve() per asset, the cache is already warm, | ||
| * so the dozens of serial ~750ms round-trips collapse into one batched call per | ||
| * graph level. |
| #ifndef AR_AYONUSDRESOLVER_PREWARM_H | ||
| #define AR_AYONUSDRESOLVER_PREWARM_H | ||
|
|
||
| #include <string> |
| * @brief C entry point for host-side (e.g. ctypes) invocation. Wraps | ||
| * AyonUsdResolverPrewarmStage. Null/empty rootAssetPath is ignored. | ||
| */ | ||
| void AyonUsdResolverPrewarm(const char *rootAssetPath); |
There was a problem hiding this comment.
Nice! I like that it's behind togglable behind an env var too. But...
Doesn't this pre-warm potentially end up loading way more data in the case of e.g. variants, unloaded payloads or even disabled prims? Because it wouldn't really take into account the scene state it may end up resolving WAY MORE usd files than what's really needed in practice.
With that it may also OPEN many more layers from disk, since it'd traverse each of them?
USD composes the stage and calls _Resolve() per asset, the cache is already warm,
so the dozens of serial ~750ms round-trips collapse into one batched call per
graph level.
This in the comments describes timing for a particular case - and admittedly one with quite a high latency, because these round trips for me seem much much smaller (even though local server for me)
| const char* legacy = std::getenv("AYON_RESOLVER_PREWARM"); | ||
| if (legacy != nullptr && (std::strcmp(legacy, "0") == 0 || std::strcmp(legacy, "false") == 0)) { |
There was a problem hiding this comment.
Do we need any 'legacy' for a 'new feature'? 🗡️
|
@BigRoy You're right, and I missed it because our performance improvement is so significant that it hid the full picture. I turned on resolver logging and counted the AYON URIs in three cases:
So 1.85x even loading everything. ~7x for proxy work. The only way to get the exact set is to compose with the real load rules and selections, but that's the exact work prewarm is trying to skip. Our crew is spread worldwide, so a lot of the team hits real per-request latency, not the sub-ms you'd get against a local server, and for them the batched warm is what keeps a heavy open from running for double-digit minutes. Maybe a solution is either (or a combination of):
|
Before USD composition pulls assets one by one, walk the layer graph via SdfLayer::GetCompositionAssetDependencies and resolve each frontier in a single batched request (AyonApi::batchResolvePathSerial), seeding the shared resolver cache. This collapses dozens of serial per-URI round-trips into a few batched ones. On by default; AYON_RESOLVER_NO_PREWARM=1 disables it. Bumps the ayon-cpp-api submodule to the perf branch (keep-alive plus batchResolvePathSerial).
I'm not really on the fence for either opt-in or opt-out, and maybe in time we find better solutions to 'load less' but still be rather optimal. I'm slightly worried if it ever touches a file that for whatever reason ends up loading 1000s of USD files but 🤷♂️ we'll see. Let's make it opt-in and have the default behavior be it disabled. Regarding payload skip. I wonder if we can somehow know the "InitialLoadSet" state when a stage would be initialized, so that if it were |
# Conflicts: # ext/ayon-cpp-api
antirotor
left a comment
antirotor
left a comment
There was a problem hiding this comment.
Built and tested in Maya - I think the only change needed is to point the ayon-cpp-api submodule to already merged commit - or better yet, release ayon-cpp-api and point the submodule to the release tag.
Regarding the number of assets that this will pull - it will be largely mitigated by introducing middle caching layer like memcached or redis.
|
This PR fixed the issue with AWS #90. I don't have specifics on hand, but looks like it was SSL. With both PRs merged, on a small scene, load times drop to about a third. From |
Changelog Description
Opening a stage against a remote AYON server is slow because USD resolves each
ayon://URI one at a time, and every resolve is a round-trip to the server. This warms the resolver cache first: it walks the layer graph and resolves each level of URIs in one batched request before USD starts composing. On a WAN link that replaces dozens of serial round-trips with a few batched ones.Merge order
This PR depends on ayon-cpp-api#47 (HTTP keep-alive and batchResolvePathSerial); merge that one first. The prewarm pass calls batchResolvePathSerial, and the submodule here is pinned to its commit, so this won't build without it. The submodule currently points at the fork commit so the branch builds and CI passes.
Benchmarking
In testing, from Cape Town via WAN to Ynput Cloud, the two changes together take a cold stage-open from about 11.7s to 2.3s, roughly 5x. On a heavier shot the same pair cut stage-open from about 33s to 4.5s. On a production character asset, the improvement was from 510s to 12s.
Additional review information
The pass lives in
prefetch/prewarm.{h,cpp}and runs fromAyonUsdResolver::_CreateDefaultContextForAsset, once per root asset. It walks the composition graph breadth-first withSdfLayer::GetCompositionAssetDependencies(), which is local layer reads only, no server calls.Each BFS frontier goes to
ResolverContextCache::batchWarm(), which resolves the frontier in one request viaAyonApi::batchResolvePathSerial()and writes the results into the process-wide cache. So when USD composes the stage and calls_Resolve()per asset, the entries are already there and the calls are local lookups.URIs get their
:SDF_FORMAT_ARGSsuffix stripped before caching, so the key matches what USD asks for during composition.The pass stays inert where it should: a no-op in static/pinning mode, and safe on a non-AYON or unreadable root, where it does nothing rather than throwing.
The process-wide cache is private to
resolverContext.cpp, so this adds a publicGetResolverGlobalCache()accessor.On by default;
AYON_RESOLVER_NO_PREWARM=1turns it off.Testing notes
PXR_PLUGINPATH_NAME, pointingAYON_SERVER_URL/AYON_API_KEYat a server you reach over a WAN or otherwise non-local link.usdcat --flatten asset.usdorusdview, with resolver logging on.AyonApi::batchResolvePathSerial(N uris)lines, one per graph frontier, and no per-URI serial resolves during composition. Note the open time.AYON_RESOLVER_NO_PREWARM=1. It should fall back to one resolve call per URI, meaning many serial round-trips, and be slower. This isolates the prewarm contribution.usdcat --flattendoes not numberFlattened_Prototype_Nprims deterministically.