Make the trust-store sidecar guard reachable, format-aware and testable

.github/workflows/offline-tests.yaml

@@ -8,6 +8,7 @@ on:
     branches: ['main']
     paths:
       - 'tests/oscap-offline/**'
+      - 'tests/stamps/**'
       - 'gpos/**'
       - 'Makefile'
       - '.github/workflows/offline-tests.yaml'
@@ -16,6 +17,7 @@ on:
     branches: ['main']
     paths:
       - 'tests/oscap-offline/**'
+      - 'tests/stamps/**'
       - 'gpos/**'
       - 'Makefile'
       - '.github/workflows/offline-tests.yaml'
@@ -64,6 +66,13 @@ jobs:
           go-version-file: 'tests/oscap-offline/go.mod'
           cache-dependency-path: 'tests/oscap-offline/go.sum'
 
+      # Hermetic: `crane` is stubbed and the fixtures are built on disk, so this
+      # needs no registry access and adds no allowed endpoints. It covers the
+      # trust-store guard's failure paths, which the daily update-ca-cert run
+      # never reaches because it only ever sees a healthy image.
+      - name: Run trust-store sidecar guard self-test
+        run: make test-stamps-selftest
+
       - name: Run offline harness
         run: make test-offline
 

.github/workflows/update-ca-cert.yaml

@@ -92,49 +92,33 @@ jobs:
         run: |
           set -euo pipefail
 
-          # CertificateAudit no longer pins any hash in the datastream: the OVAL
-          # reads each expected digest out of the stamp file the owning package
-          # ships next to the file it describes, in sha256sum format —
+          # CertificateAudit pins no hash in the datastream: the OVAL reads each
+          # expected digest out of the sidecar file the image build writes next
+          # to the file it describes, in sha256sum format —
           #   /etc/ssl/certs/.ca-certificates.crt.sha256   (always)
           #   /etc/ssl/certs/java/.cacerts.sha256          (Java images only)
           # — and compares the real file against it. Nothing in this repo needs
           # updating when either rolls upstream.
           #
-          # This step guards the premise that replaced the pins: if a future
-          # image drops a stamp file, renames it, or ships one that disagrees
-          # with the file it names, the rule starts failing on clean images.
-          # Fail loudly here — daily — instead of finding out via a red E2E run.
+          # This step guards the premise that replaced the pins. The checks
+          # themselves live in tests/stamps/run.sh so they can be run and tested
+          # outside CI (`make test-stamps`, tests/stamps/run_test.sh) rather than
+          # existing only as inline YAML; it reads the expected format out of the
+          # datastream, so the guard cannot drift from the OVAL it guards.
           #
-          # The jre image is used because it carries BOTH stamps, so a single
-          # export covers the whole rule. wolfi-base has no Java and would leave
-          # the truststore criterion unguarded.
-
-          workdir=$(mktemp -d)
-          crane export "${STEPS_IMAGE_OUTPUTS_STAMP_FULL_REF}" - | \
-            tar -C "${workdir}" -x \
-              etc/ssl/certs/ca-certificates.crt \
-              etc/ssl/certs/.ca-certificates.crt.sha256 \
-              etc/ssl/certs/java/cacerts \
-              etc/ssl/certs/java/.cacerts.sha256
-
-          # Both stamps must be present and correct. Checking each separately
-          # (rather than one combined sha256sum -c) keeps the error message
-          # specific about which trust store drifted.
-          for pair in "etc/ssl/certs:.ca-certificates.crt.sha256" "etc/ssl/certs/java:.cacerts.sha256"; do
-            dir="${pair%%:*}"
-            stamp="${pair#*:}"
-            if [ ! -f "${workdir}/${dir}/${stamp}" ]; then
-              echo "::error::${dir}/${stamp} missing from ${STAMP_IMAGE_REF}; CertificateAudit will fail on clean images"
-              exit 1
-            fi
-            if ! (cd "${workdir}/${dir}" && sha256sum -c "${stamp}"); then
-              echo "::error::${dir} contents do not match ${stamp} in ${STAMP_IMAGE_REF}; CertificateAudit will fail on clean images"
-              exit 1
-            fi
-          done
-
-          SHA=$(sha256sum "${workdir}/etc/ssl/certs/ca-certificates.crt" | cut -d' ' -f1)
-          JAVA_SHA=$(sha256sum "${workdir}/etc/ssl/certs/java/cacerts" | cut -d' ' -f1)
+          # The jre image is used because it carries BOTH sidecars the rule
+          # reads, so a single export covers them. wolfi-base has no Java and
+          # would leave the truststore criterion unguarded.
+
+          digests="${RUNNER_TEMP}/stamp-digests"
+          : > "${digests}"
+          STAMP_DIGEST_FILE="${digests}" \
+            tests/stamps/run.sh "${STEPS_IMAGE_OUTPUTS_STAMP_FULL_REF}"
+
+          # Reported for the summary below; the guard already proved each of
+          # these equals the digest recorded in the sidecar beside it.
+          SHA=$(awk '$1=="etc/ssl/certs/ca-certificates.crt"{print $2}' "${digests}")
+          JAVA_SHA=$(awk '$1=="etc/ssl/certs/java/cacerts"{print $2}' "${digests}")
           echo "sha=${SHA}" >> "$GITHUB_OUTPUT"
 
           cat >> "$GITHUB_STEP_SUMMARY" <<EOF

Makefile

@@ -45,6 +45,25 @@ test-offline-clean:
 
 .PHONY: test-offline test-offline-clean
 
+# Trust-store sidecar guard. CertificateAudit pins no digest; it reads each
+# expected value from a sidecar file the image build writes beside the trust
+# store. This target checks that premise against a real image: the sidecars
+# exist, agree with the files they name, and are formatted so the OVAL's own
+# regex (read from the datastream, not copied) matches them. The daily
+# update-ca-cert workflow runs the same script.
+#
+# Override the images with STAMP_IMAGES="ref [ref ...]".
+test-stamps:
+	@tests/stamps/run.sh
+
+# Exercise the guard's failure paths against synthetic images, with `crane`
+# stubbed so no registry or network is needed. These are the paths a green
+# daily run never reaches.
+test-stamps-selftest:
+	@tests/stamps/run_test.sh
+
+.PHONY: test-stamps test-stamps-selftest
+
 # Extract the XCCDF Benchmark block from the datastream and diff it
 # against BASE_REF (default: origin/main). STIGViewer v3 loads the full
 # datastream directly, so this target's job is to surface content drift

README.md

@@ -165,3 +165,35 @@ Adding a fixture:
 - Create `tests/e2e/fixtures/<name>/expected.txt` listing each rule
   ID you want to assert on, one per line, as
   `<rule-id>=<expected-result>`.
+
+### Trust-store sidecar guard
+
+Both tiers above scan images this repository builds. `CertificateAudit`
+additionally depends on something no fixture can assert: that *real*
+images still ship the trust-store checksum sidecars it reads, in the
+format its regexes expect. `make test-stamps` checks that premise
+against a live image, reading the expected patterns out of the
+datastream so the guard cannot drift from the definition it guards. The
+daily `.github/workflows/update-ca-cert.yaml` run invokes the same
+script, so an upstream change fails there rather than surfacing later as
+an unexplained failure on clean images.
+
+```bash
+make test-stamps            # default: cgr.dev/chainguard/jre:latest
+make test-stamps-selftest   # the guard's own failure paths; hermetic
+```
+
+The self-test stubs `crane` and builds its images on disk, so it needs
+no registry or network, and it runs on PRs via the offline workflow. It
+covers the failure paths specifically because a green daily run only
+ever sees a healthy image.
+
+Pass image references to widen coverage — `crane` uses the same registry
+credentials as `docker`. The `/kaniko` bundle-copy criteria are only
+reachable on a private image, so the default run does not cover them and
+says so; see
+[docs/certificate-audit.md](./docs/certificate-audit.md#covering-the-kaniko-criteria).
+
+```bash
+make test-stamps STAMP_IMAGES="cgr.dev/chainguard/jre:latest cgr.dev/chainguard-private/kaniko:latest"
+```

docs/certificate-audit.md

@@ -280,6 +280,64 @@ Two cautions when editing that matrix:
   variables, so a fixture blob can exist with no row referencing it and the
   suite still passes. Check the subtest count, not just the diff.
 
+### Guarding the premise
+
+Those fixtures test the rule against images this repository builds. They cannot
+tell you that a *real* image still ships sidecars in the shape the rule expects
+— which is the assumption the whole rule now rests on, and which upstream can
+change without warning. `tests/stamps/run.sh` covers that: for each trust store
+in an image it checks the sidecar exists, agrees with the file it names, and
+matches the regex the OVAL will apply. The pattern is read out of the datastream
+rather than copied, so the guard cannot drift from the definition it guards.
+
+    make test-stamps                     # default: cgr.dev/chainguard/jre:latest
+    make test-stamps-selftest            # failure paths, hermetic, no registry
+
+`jre` is the default because it is a public image carrying both the system and
+Java sidecars. The daily `update-ca-cert` workflow runs the same script, so a
+sidecar that disappears or changes format upstream fails there rather than
+surfacing later as a mysterious `CertificateAudit` failure on clean images.
+
+`run_test.sh` is where the failure paths live, with `crane` stubbed so the cases
+need no registry or network. It matters because a green daily run only ever sees
+a healthy image, so nothing else exercises the diagnostics.
+
+#### Covering the /kaniko criteria
+
+The `/kaniko` bundle copy exists only on `cgr.dev/chainguard-private/kaniko`, so
+the default run does not reach it and says so. Anyone with access to that image
+can cover it locally by naming it.
+
+`crane` reads the same credentials as `docker`, and `cgr.dev` is served by the
+`cgr` credential helper, which needs a token issued for the `cgr.dev` audience
+specifically:
+
+    chainctl auth login --audience=cgr.dev
+
+A plain `chainctl auth login` is not enough — that token is scoped to the
+console API, so `chainctl auth status` reports `Valid: True` while the pull
+still fails with `No matching credentials were found for "cgr.dev"`. The
+credential helper itself is usually already wired up (`chainctl auth
+configure-docker` will say so); the audience is the part that goes missing.
+
+    tests/stamps/run.sh cgr.dev/chainguard/jre:latest \
+                        cgr.dev/chainguard-private/kaniko:latest
+
+    # or, equivalently
+    make test-stamps STAMP_IMAGES="cgr.dev/chainguard/jre:latest cgr.dev/chainguard-private/kaniko:latest"
+
+A run reports what it did not reach, so passing that ref drops the `/kaniko`
+caveat from the closing note instead of printing it.
+
+This is deliberately not automated. Doing so would need two additions to the
+workflow's trust surface, not one: a credential for the private registry, and a
+second accepted signer identity, because that image is signed by
+`chainguard-dev/stereo/.github/workflows/release-containers.yaml` rather than
+the `chainguard-images/images/*` identity the workflow requires. The copy is
+currently byte-identical to its system bundle, so the drift being guarded
+against is remote; the trade was judged not worth it for now. Revisit if the
+`/kaniko` copy ever starts diverging, or gains a sidecar of its own.
+
 ## Known gaps
 
 - apko also stamps `var/lib/ecs/deps/execute-command/certs/tls-ca-bundle.pem`
@@ -293,3 +351,7 @@ Two cautions when editing that matrix:
   containing two matching lines silently uses the first.
 - Deleting `/etc/ssl/certs/java/cacerts` outright satisfies the Java `OR` via
   `tst:5`.
+- No automated run guards the `/kaniko` criteria against a real image; the
+  daily workflow inspects only public images. It is coverable on demand — see
+  [Covering the /kaniko criteria](#covering-the-kaniko-criteria) — and deferred
+  rather than declined.