Skip to content

[ENG-603] Fix caching in care backend build actions #3696

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
tellmeY18 wants to merge 5 commits into
base: develop
Choose a base branch
from
Open

[ENG-603] Fix caching in care backend build actions #3696

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
9d9a3ea
ci: switch Docker cache to GHCR registry with weekly rotation + plugi…
tellmeY18 Jun 26, 2026
9a4b733
Update .github/scripts/resolve_plugins.py
tellmeY18 Jun 29, 2026
76ddd40
Set default value for PLUGIN_RESOLVED_HASH in Dockerfile
vigneshhari Jul 1, 2026
ba2e22f
Set default value for PLUGIN_RESOLVED_HASH argument
vigneshhari Jul 1, 2026
7faf1f7
Address review comments on plugin cache busting
Jul 4, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
45 changes: 45 additions & 0 deletions .github/scripts/resolve_plugins.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,45 @@
#!/usr/bin/env python3
import hashlib
import json
import os
import subprocess
import sys

plugs_raw = os.environ.get("ADDITIONAL_PLUGS", "[]")
try:
plugs = json.loads(plugs_raw)
except json.JSONDecodeError:
sys.stdout.write("no-plugins")
sys.exit(0)

if not plugs:
sys.stdout.write("no-plugins")
sys.exit(0)

resolved = []
for p in plugs:
ver = p.get("version", "@main")
pkg = p.get("package_name")
if not pkg:
sys.stderr.write(f"plugin entry {p!r} is missing 'package_name'\n")
sys.exit(1)
if ver.startswith("@"):
ref = ver[1:]
git_url = pkg.removeprefix("git+")
try:
out = subprocess.run( # noqa: S603
["git", "ls-remote", git_url, ref], # noqa: S607
check=False,
capture_output=True,
text=True,
timeout=15,
)
sha = out.stdout.split()[0] if out.stdout else ref
except Exception:
sha = ref
Comment thread
tellmeY18 marked this conversation as resolved.
Outdated
resolved.append(f"{pkg}@{sha}")
else:
resolved.append(f"{pkg}{ver}")
Comment on lines +26 to +45

@coderabbitai coderabbitai Bot Jun 26, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟡 Minor

🧩 Analysis chain

🏁 Script executed:

python3 - <<'PY'
from pathlib import Path
p = Path('.github/scripts/resolve_plugins.py')
print(p.exists(), p)
print(p.read_text())
PY

Repository: ohcnetwork/care

Length of output: 1395

🏁 Script executed:

python3 - <<'PY'
from pathlib import Path
p = Path('.github/scripts/resolve_plugins.py')
print(p.exists(), p)
print(p.read_text())
PY

Repository: ohcnetwork/care

Length of output: 1395

Avoid falling back to the raw ref here. When git ls-remote fails or returns nothing, sha becomes ref, so @main hashes like pkg@main and stops busting the cache when the branch moves. Surface the failure instead of quietly pretending it resolved.

Also, p.get("version", "@main") does not cover "version": null; that still reaches ver.startswith("@") and crashes. Use p.get("version") or "@main" if null should mean default.

🧰 Tools
🪛 ast-grep (0.44.0)

[error] 29-35: Command coming from incoming request
Context: subprocess.run( # noqa: S603
["git", "ls-remote", git_url, ref], # noqa: S607
check=False,
capture_output=True,
text=True,
timeout=15,
)
Note: [CWE-78] Improper Neutralization of Special Elements used in an OS Command ('OS Command Injection').

(subprocess-from-request)

🪛 Ruff (0.15.18)

[warning] 38-38: Do not catch blind exception: Exception

(BLE001)

🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In @.github/scripts/resolve_plugins.py around lines 26 - 42, The plugin ref
resolution in resolve_plugins.py is silently falling back to the raw branch/ref
when git ls-remote fails or returns no output, which makes values like pkg@main
stop changing and hides resolution problems; update the logic around the ref
handling to surface the failure instead of assigning sha from ref. Also fix the
version defaulting in the package version lookup so a null version is treated
like missing input by using the same default path as the versionless case, and
make sure the ver.startswith("@") branch in the resolver no longer crashes on
null values.

Source: Linters/SAST tools


h = hashlib.sha256(json.dumps(sorted(resolved)).encode()).hexdigest()[:16]
print(h, end="") # noqa: T201
66 changes: 49 additions & 17 deletions .github/workflows/deploy.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -73,18 +73,17 @@ jobs:
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}

- name: Cache Docker layers
uses: actions/cache@v4
with:
path: ${{ runner.temp }}/.buildx-cache
key: ${{ runner.os }}-${{ runner.arch }}-buildx-prod-${{ hashFiles('Pipfile.lock', 'docker/prod.Dockerfile') }}
restore-keys: |
${{ runner.os }}-${{ runner.arch }}-buildx-prod-
- name: Compute cache tag
id: cache-tag
run: |
echo "tag=buildcache-${{ env.PLATFORM_PAIR }}-$(date +%G-W%V)" >> "$GITHUB_OUTPUT"

- name: Create new cache
- name: Resolve plugin cache buster
id: plugin-hash
run: |
mkdir -p ${{ runner.temp }}/.buildx-cache
mkdir -p ${{ runner.temp }}/.buildx-cache-new
echo "resolved_hash=$(python3 .github/scripts/resolve_plugins.py)" >> "$GITHUB_OUTPUT"
env:
ADDITIONAL_PLUGS: ${{ env.ADDITIONAL_PLUGS }}

- name: Build and push by digest
id: build
Expand All @@ -98,8 +97,9 @@ jobs:
build-args: |
APP_VERSION=${{ github.sha }}
ADDITIONAL_PLUGS=${{ env.ADDITIONAL_PLUGS }}
cache-from: type=local,src=${{ runner.temp }}/.buildx-cache
cache-to: type=local,dest=${{ runner.temp }}/.buildx-cache-new,mode=max
PLUGIN_RESOLVED_HASH=${{ steps.plugin-hash.outputs.resolved_hash }}
cache-from: type=registry,ref=ghcr.io/${{ github.repository }}:${{ steps.cache-tag.outputs.tag }}
cache-to: type=registry,ref=ghcr.io/${{ github.repository }}:${{ steps.cache-tag.outputs.tag }},mode=max

- name: Export digest
run: |
Expand All @@ -115,11 +115,6 @@ jobs:
if-no-files-found: error
retention-days: 1

- name: Move cache
run: |
rm -rf ${{ runner.temp }}/.buildx-cache
mv ${{ runner.temp }}/.buildx-cache-new ${{ runner.temp }}/.buildx-cache

merge-manifests:
needs: build
name: Merge & Push Manifests
Expand Down Expand Up @@ -186,6 +181,43 @@ jobs:
run: |
echo "Release ${{ github.sha }} is ready to be deployed to production"

prune-cache:
if: github.ref == 'refs/heads/develop' && github.repository == 'ohcnetwork/care'
needs: merge-manifests
runs-on: ubuntu-24.04
permissions:
packages: write
steps:
- name: Prune stale buildcache versions
env:
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
run: |
OWNER="${{ github.repository_owner }}"
PACKAGE_NAME="${GITHUB_REPOSITORY#*/}"
CUTOFF=$(date -d "4 weeks ago" +%G-W%V)

gh api "/orgs/$OWNER/packages/container/$PACKAGE_NAME/versions?per_page=100" \
--paginate \
--jq '.[] | select(any(.metadata.container.tags[]; startswith("buildcache-")))' \
| python3 -c "
import json, re, sys
cutoff = '$CUTOFF'
for line in sys.stdin:
line = line.strip()
if not line:
continue
v = json.loads(line)
for tag in v['metadata']['container']['tags']:
m = re.match(r'^buildcache-.+-(\d{4}-W\d{2})$', tag)
if m and m.group(1) < cutoff:
print(v['id'], tag)
break
" | while read -r id tag; do
echo "Pruning $tag (version $id)"
gh api --method DELETE "/orgs/$OWNER/packages/container/$PACKAGE_NAME/versions/$id" \
--silent || echo " (skipped)"
done

# deploy-staging-egov:
# needs: merge-manifests
# if: github.ref == 'refs/heads/develop'
Expand Down
58 changes: 21 additions & 37 deletions .github/workflows/reusable-test.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,39 +21,37 @@ jobs:
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v3

- name: Restore Docker layers cache
id: cache-restore
uses: actions/cache/restore@v4
- name: Login to GitHub Container Registry
uses: docker/login-action@v3
with:
path: |
${{ runner.temp }}/.buildx-cache
${{ runner.temp }}/.buildx-mounted-cache
key: ${{ runner.os }}-${{ runner.arch }}-buildx-dev-${{ hashFiles('Pipfile.lock', 'docker/dev.Dockerfile') }}
restore-keys: |
${{ runner.os }}-${{ runner.arch }}-buildx-dev-
registry: ghcr.io
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}

- name: inject cache into docker
uses: reproducible-containers/buildkit-cache-dance@v3.3.0
with:
cache-map: |
{
"${{ runner.temp }}/.buildx-mounted-cache": "/root/.cache/pip"
}
skip-extraction: ${{ steps.cache-restore.outputs.cache-hit }}
- name: Compute cache tag
run: |
echo "CACHE_TAG=buildcache-dev-$(date +%G-W%V)" >> "$GITHUB_ENV"

- name: Create new cache
- name: Resolve plugin cache buster
id: plugin-hash
run: |
mkdir -p ${{ runner.temp }}/.buildx-cache
mkdir -p ${{ runner.temp }}/.buildx-cache-new
mkdir -p ${{ runner.temp }}/.buildx-mounted-cache
echo "resolved_hash=$(python3 .github/scripts/resolve_plugins.py)" >> "$GITHUB_OUTPUT"
env:
ADDITIONAL_PLUGS: ${{ env.ADDITIONAL_PLUGS }}

- name: Build images
run: |
CACHE_TO=""
if [[ '${{ github.event_name }}' == 'push' ]]; then
CACHE_TO="--cache-to=type=registry,ref=ghcr.io/${{ github.repository }}:$CACHE_TAG,mode=max"
fi
docker buildx build \
--file docker/dev.Dockerfile \
--tag care_local \
--cache-from=type=local,src=${{ runner.temp }}/.buildx-cache \
--cache-to=type=local,dest=${{ runner.temp }}/.buildx-cache-new,mode=max \
--build-arg PLUGIN_RESOLVED_HASH=${{ steps.plugin-hash.outputs.resolved_hash }} \
--build-arg ADDITIONAL_PLUGS=${{ env.ADDITIONAL_PLUGS }} \
Comment thread
greptile-apps[bot] marked this conversation as resolved.
Outdated
--cache-from=type=registry,ref=ghcr.io/${{ github.repository }}:$CACHE_TAG \
$CACHE_TO \
Comment thread
coderabbitai[bot] marked this conversation as resolved.
--platform linux/arm64 \
--load \
.
Expand Down Expand Up @@ -81,20 +79,6 @@ jobs:
env:
CODECOV_TOKEN: ${{ secrets.CODECOV_TOKEN }}

- name: Move cache
run: |
rm -rf ${{ runner.temp }}/.buildx-cache
mv ${{ runner.temp }}/.buildx-cache-new ${{ runner.temp }}/.buildx-cache

- name: Save Docker layers cache
if: ${{ github.event_name == 'push' && github.ref == 'refs/heads/develop' && steps.cache-restore.outputs.cache-hit != 'true' }}
uses: actions/cache/save@v4
with:
path: |
${{ runner.temp }}/.buildx-cache
${{ runner.temp }}/.buildx-mounted-cache
key: ${{ runner.os }}-${{ runner.arch }}-buildx-dev-${{ hashFiles('Pipfile.lock', 'docker/dev.Dockerfile') }}

# Upload dummy db as artifact so it can be used to speed up frontend tests
- name: Dump db
if: ${{ github.event_name == 'push' && github.ref == 'refs/heads/develop' }}
Expand Down
1 change: 1 addition & 0 deletions docker/dev.Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,7 @@ RUN --mount=type=cache,target=/root/.cache/pip pip install pipenv==2025.1.1
COPY Pipfile Pipfile.lock $APP_HOME/
RUN --mount=type=cache,target=/root/.cache/pip pipenv install --system --categories "packages dev-packages docs"

ARG PLUGIN_RESOLVED_HASH
ARG ADDITIONAL_PLUGS=""
ENV ADDITIONAL_PLUGS=$ADDITIONAL_PLUGS

Expand Down
1 change: 1 addition & 0 deletions docker/prod.Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,7 @@ RUN pipenv install --deploy --categories "packages"
COPY plugs/ $APP_HOME/plugs/
COPY install_plugins.py plug_config.py $APP_HOME/

ARG PLUGIN_RESOLVED_HASH
ARG ADDITIONAL_PLUGS=""
ENV ADDITIONAL_PLUGS=$ADDITIONAL_PLUGS
RUN python3 $APP_HOME/install_plugins.py
Comment thread
coderabbitai[bot] marked this conversation as resolved.
Outdated
Expand Down
Loading