ci: make build cache restore resilient to eviction (#16275)

# Overview

Downstream CI jobs fail when re-running individual jobs after the build
cache has been evicted from GitHub's 10GB repo-wide LRU cache. For
example, [this
run](https://github.com/payloadcms/payload/actions/runs/24259697364/job/71108809903?pr=15268#step:5:22)
had 4 jobs fail at "Restore build" because the cache was evicted ~43
hours after the original build.

## Time Savings

This replaces the fixed 120s `sleep` propagation delay and hard-failing
`fail-on-cache-miss: true` with a polling + fallback approach that
self-heals on cache miss. **This change will save 2 mins per run.**

## Key Changes

- **New `restore-build` input on the setup action**
- When `true`, the action polls for the build cache using `gh cache
list` (10s intervals, 120s timeout), restores it if found, or falls back
to `pnpm install && pnpm run build:all` if not. This replaces
`pnpm-run-install: false`, `pnpm-restore-cache: false`,
`cache-propagation-delay: 120`, and the separate `Restore build` step
that every downstream job previously had.

- **Simplified downstream jobs**
- All 8 downstream jobs (`tests-unit`, `tests-types`, `tests-int`,
`e2e-prep`, `tests-e2e`, `build-and-test-templates`,
`tests-type-generation`, `analyze`) go from ~10 lines of setup + cache
restore boilerplate to `restore-build: true`.

- **Removed `cache-propagation-delay` input**
- The fixed 120s sleep is no longer needed. Polling finds the cache as
soon as it's available (typically instantly), and fallback handles the
miss case.

## Design Decisions

**Polling over fixed sleep:** The old 120s delay was wasted time on the
happy path (propagation is effectively instant) and didn't help when the
cache was evicted entirely. Polling with `gh cache list` finds the cache
as soon as it propagates, and the timeout triggers a fallback build
instead of a hard failure.

**`gh cache list` over `actions/cache/restore` with `lookup-only`:**
`lookup-only` is a step-level action that can't be called in a bash
loop. `gh cache list` is a CLI command available on all runners, works
in a loop, and requires no extra extensions. Uses exact key match via jq
filter to avoid prefix-match false positives.

**Fallback builds with warm pnpm store:** The pnpm store cache is
restored regardless of `restore-build` mode, so if the fallback build
triggers, `pnpm install` has a warm store rather than starting cold.

**Error handling:** If `gh cache list` fails (rate limit, auth,
network), the loop breaks immediately and falls back to a full build
rather than silently polling for 120s.

## Overall Flow

```mermaid
sequenceDiagram
    participant B as Build Job
    participant C as GitHub Cache
    participant D as Downstream Job
    participant S as Setup Action

    B->>C: Save build cache (key: SHA)
    D->>S: restore-build: true
    S->>S: Restore pnpm store cache
    loop Every 10s (up to 120s)
        S->>C: gh cache list (exact key match)
        C-->>S: Found / Not found
    end
    alt Cache found
        S->>C: actions/cache/restore
        C-->>S: Build artifacts
    else Cache not found (evicted or timeout)
        S->>S: pnpm install && pnpm run build:all
    end
```

## References / Links

- [actions/cache#1710](actions/cache#1710) —
original cache propagation delay issue
- [gh cache list docs](https://cli.github.com/manual/gh_cache_list)
- [actions/cache/restore
README](https://github.com/actions/cache/blob/main/restore/README.md)

Author: denolfe

.github/actions/setup/action.yml

@@ -15,9 +15,9 @@ inputs:
     default: 'true'
   pnpm-install-cache-key:
     description: The cache key override for the pnpm install cache
-  cache-propagation-delay:
-    description: Seconds to wait after setup for cache propagation (workaround for https://github.com/actions/cache/issues/1710)
-    default: '0'
+  restore-build:
+    description: Whether to restore the build cache (polls for availability, falls back to full build on miss)
+    default: 'false'
 
 outputs:
   pnpm-install-cache-key:
@@ -92,7 +92,7 @@ runs:
         echo "PNPM_INSTALL_CACHE_KEY=$PNPM_INSTALL_CACHE_KEY" >> $GITHUB_ENV
 
     - name: Restore pnpm install cache
-      if: ${{ inputs.pnpm-restore-cache == 'true' }}
+      if: ${{ inputs.pnpm-restore-cache == 'true' || inputs.restore-build == 'true' }}
       uses: actions/cache@27d5ce7f107fe9357f9df03efb73ab90386fccae # v5.0.5
       with:
         path: ${{ env.STORE_PATH }}
@@ -102,7 +102,7 @@ runs:
           pnpm-store-
 
     - name: Run pnpm install
-      if: ${{ inputs.pnpm-run-install == 'true' }}
+      if: ${{ inputs.pnpm-run-install == 'true' && inputs.restore-build != 'true' }}
       shell: bash
       run: pnpm install
 
@@ -112,9 +112,54 @@ runs:
       shell: bash
       id: compute-output
 
-    - name: Wait for cache propagation
-      if: ${{ inputs.cache-propagation-delay != '0' }}
+    - name: Poll for build cache
+      if: ${{ inputs.restore-build == 'true' }}
       shell: bash
+      env:
+        GH_TOKEN: ${{ github.token }}
       run: |
-        echo "Waiting ${{ inputs.cache-propagation-delay }}s for cache propagation..."
-        sleep ${{ inputs.cache-propagation-delay }}
+        CACHE_KEY="${{ github.sha }}"
+        POLL_INTERVAL=10
+        TIMEOUT=120
+        ELAPSED=0
+        CACHE_FOUND=false
+
+        echo "Polling for build cache key: $CACHE_KEY"
+
+        while [ $ELAPSED -lt $TIMEOUT ]; do
+          if ! RESULT=$(gh cache list --repo "$GITHUB_REPOSITORY" --key "$CACHE_KEY" --json key --jq "[.[] | select(.key == \"$CACHE_KEY\")] | length"); then
+            echo "::warning::gh cache list failed, falling back to full build"
+            break
+          fi
+          if [ "$RESULT" -gt 0 ]; then
+            echo "Cache found after ${ELAPSED}s"
+            CACHE_FOUND=true
+            break
+          fi
+          echo "Cache not found yet (${ELAPSED}s elapsed), retrying in ${POLL_INTERVAL}s..."
+          sleep $POLL_INTERVAL
+          ELAPSED=$((ELAPSED + POLL_INTERVAL))
+        done
+
+        if [ "$CACHE_FOUND" = "false" ]; then
+          echo "::warning::Build cache not found after ${TIMEOUT}s — will run full build as fallback"
+        fi
+
+        echo "CACHE_FOUND=$CACHE_FOUND" >> $GITHUB_ENV
+
+    - name: Restore build cache
+      if: ${{ inputs.restore-build == 'true' && env.CACHE_FOUND == 'true' }}
+      uses: actions/cache/restore@27d5ce7f107fe9357f9df03efb73ab90386fccae # v5.0.5
+      with:
+        path: ./*
+        key: ${{ github.sha }}
+
+    - name: Fallback — install and build
+      if: ${{ inputs.restore-build == 'true' && env.CACHE_FOUND != 'true' }}
+      shell: bash
+      run: |
+        echo "::warning::Build cache miss — running full build as fallback"
+        pnpm install
+        pnpm run build:all
+      env:
+        DO_NOT_TRACK: 1

.github/workflows/main.yml

@@ -103,16 +103,7 @@ jobs:
       - name: Node setup
         uses: ./.github/actions/setup
         with:
-          pnpm-run-install: false
-          pnpm-restore-cache: false # Full build is restored below
-          cache-propagation-delay: 120 # https://github.com/actions/cache/issues/1710
-
-      - name: Restore build
-        uses: actions/cache@27d5ce7f107fe9357f9df03efb73ab90386fccae # v5.0.5
-        with:
-          path: ./*
-          key: ${{ github.sha }}
-          fail-on-cache-miss: true
+          restore-build: true
 
       - name: Unit Tests
         run: pnpm test:unit
@@ -129,16 +120,7 @@ jobs:
       - name: Node setup
         uses: ./.github/actions/setup
         with:
-          pnpm-run-install: false
-          pnpm-restore-cache: false # Full build is restored below
-          cache-propagation-delay: 120 # https://github.com/actions/cache/issues/1710
-
-      - name: Restore build
-        uses: actions/cache@27d5ce7f107fe9357f9df03efb73ab90386fccae # v5.0.5
-        with:
-          path: ./*
-          key: ${{ github.sha }}
-          fail-on-cache-miss: true
+          restore-build: true
 
       - name: Types Tests
         run: pnpm test:types --target '>=5.7'
@@ -198,16 +180,7 @@ jobs:
       - name: Node setup
         uses: ./.github/actions/setup
         with:
-          pnpm-run-install: false
-          pnpm-restore-cache: false # Full build is restored below
-          cache-propagation-delay: 120 # https://github.com/actions/cache/issues/1710
-
-      - name: Restore build
-        uses: actions/cache@27d5ce7f107fe9357f9df03efb73ab90386fccae # v5.0.5
-        with:
-          path: ./*
-          key: ${{ github.sha }}
-          fail-on-cache-miss: true
+          restore-build: true
 
       - name: Start LocalStack
         run: pnpm docker:start
@@ -266,16 +239,7 @@ jobs:
       - name: Node setup
         uses: ./.github/actions/setup
         with:
-          pnpm-run-install: false
-          pnpm-restore-cache: false
-          cache-propagation-delay: 120 # https://github.com/actions/cache/issues/1710
-
-      - name: Restore build
-        uses: actions/cache@27d5ce7f107fe9357f9df03efb73ab90386fccae # v5.0.5
-        with:
-          path: ./*
-          key: ${{ github.sha }}
-          fail-on-cache-miss: true
+          restore-build: true
 
       - name: Prepare prod test environment
         run: pnpm prepare-run-test-against-prod:ci
@@ -307,16 +271,7 @@ jobs:
       - name: Node setup
         uses: ./.github/actions/setup
         with:
-          pnpm-run-install: false
-          pnpm-restore-cache: false # Full build is restored below
-          cache-propagation-delay: 120 # https://github.com/actions/cache/issues/1710
-
-      - name: Restore build
-        uses: actions/cache@27d5ce7f107fe9357f9df03efb73ab90386fccae # v5.0.5
-        with:
-          path: ./*
-          key: ${{ github.sha }}
-          fail-on-cache-miss: true
+          restore-build: true
 
       - name: Restore prepared test environment
         uses: actions/cache/restore@27d5ce7f107fe9357f9df03efb73ab90386fccae # v5.0.5
@@ -447,16 +402,7 @@ jobs:
       - name: Node setup
         uses: ./.github/actions/setup
         with:
-          pnpm-run-install: false
-          pnpm-restore-cache: false # Full build is restored below
-          cache-propagation-delay: 120 # https://github.com/actions/cache/issues/1710
-
-      - name: Restore build
-        uses: actions/cache@27d5ce7f107fe9357f9df03efb73ab90386fccae # v5.0.5
-        with:
-          path: ./*
-          key: ${{ github.sha }}
-          fail-on-cache-miss: true
+          restore-build: true
 
       - name: Start database
         id: db
@@ -522,16 +468,7 @@ jobs:
       - name: Node setup
         uses: ./.github/actions/setup
         with:
-          pnpm-run-install: false
-          pnpm-restore-cache: false # Full build is restored below
-          cache-propagation-delay: 120 # https://github.com/actions/cache/issues/1710
-
-      - name: Restore build
-        uses: actions/cache@27d5ce7f107fe9357f9df03efb73ab90386fccae # v5.0.5
-        with:
-          path: ./*
-          key: ${{ github.sha }}
-          fail-on-cache-miss: true
+          restore-build: true
 
       - name: Generate Payload Types
         run: pnpm dev:generate-types fields
@@ -585,16 +522,7 @@ jobs:
       - name: Node setup
         uses: ./.github/actions/setup
         with:
-          pnpm-run-install: false
-          pnpm-restore-cache: false # Full build is restored below
-          cache-propagation-delay: 120 # https://github.com/actions/cache/issues/1710
-
-      - name: Restore build
-        uses: actions/cache@27d5ce7f107fe9357f9df03efb73ab90386fccae # v5.0.5
-        with:
-          path: ./*
-          key: ${{ github.sha }}
-          fail-on-cache-miss: true
+          restore-build: true
 
       - run: pnpm run build:bundle-for-analysis # Esbuild packages that haven't already been built in the build step for the purpose of analyzing bundle size
         env: