ci(embedding-sidecar): ride out huggingface outages with backoff retries#1087
Merged
Conversation
The embedding-sidecar image pre-downloads the nomic-embed model from huggingface.co at build time. huggingface has sustained gateway-timeout windows; the old 5x15s (~75s) flat retry couldn't outlast one, and because a build-time fetch failure aborts the whole "Build containers" step, a single huggingface wobble red-lined EVERY open PR's CI at once (observed today across #1057 and #1085 with identical `Gateway timeout ... nomic-embed-text-v1.5` logs). Widen both the npm-install and model-prebuild retry loops to 8 attempts with exponential backoff (15s -> 120s cap, ~10 min total) so a passing build no longer hinges on huggingface being reachable for one ~minute window. Behaviour is unchanged when the network is healthy (first attempt succeeds). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rm8xqAcBsnaYWZkrp5KAMA
✅ Deploy Preview for golden-caramel-d2c3a7 canceled.
|
FreegleGeeks
added
CI: Running
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The
embedding-sidecarimage pre-downloads thenomic-embed-text-v1.5model from huggingface.co at build time. huggingface has sustained gateway-timeout windows, and the old 5×15s (~75s) flat retry can't outlast one. Because a build-time fetch failure aborts the whole "Build containers" step, a single huggingface wobble red-lines every open PR's CI at once.Observed today across two unconnected PRs with the identical failure:
build-and-testjob 27334) and feat(digest): sink already-seen posts in the daily digest and browse feed #1085 (job 27337) both aborted "Build containers" withError: Gateway timeout ... huggingface.co/nomic-ai/nomic-embed-text-v1.5/resolve/main/tokenizer.json→model prebuild failed after 5 attempts. Neither commit touches the embedding sidecar, and the test suites never ran.Fix
Widen both the npm-install and model-prebuild retry loops to 8 attempts with exponential backoff (15s → 120s cap, ~10 min total window) instead of 5×15s. A passing build no longer hinges on huggingface being reachable for one ~minute window. Behaviour is unchanged when the network is healthy — the first attempt succeeds.
Test Plan
docker/embedding-sidecar/Dockerfilelocally with the change: build succeeds,Model cached.on the first attempt (backoff loop shell syntax verified, model fetch works).🤖 Generated with Claude Code
https://claude.ai/code/session_01Rm8xqAcBsnaYWZkrp5KAMA