From 33a39abdffe40efb169579e0648cc5834cd6bfa9 Mon Sep 17 00:00:00 2001 From: Steve Beattie Date: Thu, 20 Aug 2026 12:29:44 -0700 Subject: [PATCH 1/3] Make the trust-store sidecar guard reachable, format-aware and testable MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The daily guard that checks CertificateAudit's premise had two holes, both in code paths a healthy image never reaches, so neither could show up in a green run. First, the missing-sidecar diagnostic was unreachable. Extraction named each file individually, and `tar -x ` exits 2 when a member is absent from the archive, so under `set -euo pipefail` the step died on tar's "Not found in archive" before reaching the branch that says which trust store lost its sidecar — the exact diagnosis the guard exists to produce. Extraction now asks for the containing directories, and absence is reported by the per-file checks. Second, the guard checked the digest but not the format. The OVAL matches each sidecar with a regex; `sha256sum -c` is far more permissive. A sidecar naming an absolute path verifies fine under `sha256sum -c` and fails the rule, so a format change upstream would pass this guard and start failing CertificateAudit on clean images — precisely what the guard is meant to catch first. The patterns are now read out of the datastream and applied with `grep -P`, so the guard cannot drift from the OVAL it guards. The checks move from inline YAML to tests/stamps/run.sh, which makes them runnable outside CI (`make test-stamps`) and lets the failure paths be tested at all. tests/stamps/run_test.sh covers them against synthetic images with `crane` stubbed — hermetic, no registry, no network — and runs on PRs via the offline workflow. Confirmed the cases fail when each bug is reintroduced: removing the pattern assertion fails exactly the absolute-path case. The guard also now refuses to report success if it verified nothing, and prints what it does not cover, so a green run is not read as covering the whole rule. Two things remain uncovered and are named in that output: /kaniko/ssl/certs, which exists only on a private image this workflow has no credentials for, and the ECS bundle, which the image build stamps but CertificateAudit does not read. Verified: shellcheck clean; zizmor no findings; actionlint issue count unchanged from main; `make test-stamps` passes against the live jre image and `make test-stamps-selftest` passes all nine cases. Co-Authored-By: Claude Opus 5 --- .github/workflows/offline-tests.yaml | 9 ++ .github/workflows/update-ca-cert.yaml | 58 +++----- Makefile | 19 +++ tests/stamps/run.sh | 192 ++++++++++++++++++++++++++ tests/stamps/run_test.sh | 144 +++++++++++++++++++ 5 files changed, 385 insertions(+), 37 deletions(-) create mode 100755 tests/stamps/run.sh create mode 100755 tests/stamps/run_test.sh diff --git a/.github/workflows/offline-tests.yaml b/.github/workflows/offline-tests.yaml index 681b27c..b60d07e 100644 --- a/.github/workflows/offline-tests.yaml +++ b/.github/workflows/offline-tests.yaml @@ -8,6 +8,7 @@ on: branches: ['main'] paths: - 'tests/oscap-offline/**' + - 'tests/stamps/**' - 'gpos/**' - 'Makefile' - '.github/workflows/offline-tests.yaml' @@ -16,6 +17,7 @@ on: branches: ['main'] paths: - 'tests/oscap-offline/**' + - 'tests/stamps/**' - 'gpos/**' - 'Makefile' - '.github/workflows/offline-tests.yaml' @@ -64,6 +66,13 @@ jobs: go-version-file: 'tests/oscap-offline/go.mod' cache-dependency-path: 'tests/oscap-offline/go.sum' + # Hermetic: `crane` is stubbed and the fixtures are built on disk, so this + # needs no registry access and adds no allowed endpoints. It covers the + # trust-store guard's failure paths, which the daily update-ca-cert run + # never reaches because it only ever sees a healthy image. + - name: Run trust-store sidecar guard self-test + run: make test-stamps-selftest + - name: Run offline harness run: make test-offline diff --git a/.github/workflows/update-ca-cert.yaml b/.github/workflows/update-ca-cert.yaml index 77612c3..2bd1d7b 100644 --- a/.github/workflows/update-ca-cert.yaml +++ b/.github/workflows/update-ca-cert.yaml @@ -92,49 +92,33 @@ jobs: run: | set -euo pipefail - # CertificateAudit no longer pins any hash in the datastream: the OVAL - # reads each expected digest out of the stamp file the owning package - # ships next to the file it describes, in sha256sum format — + # CertificateAudit pins no hash in the datastream: the OVAL reads each + # expected digest out of the sidecar file the image build writes next + # to the file it describes, in sha256sum format — # /etc/ssl/certs/.ca-certificates.crt.sha256 (always) # /etc/ssl/certs/java/.cacerts.sha256 (Java images only) # — and compares the real file against it. Nothing in this repo needs # updating when either rolls upstream. # - # This step guards the premise that replaced the pins: if a future - # image drops a stamp file, renames it, or ships one that disagrees - # with the file it names, the rule starts failing on clean images. - # Fail loudly here — daily — instead of finding out via a red E2E run. + # This step guards the premise that replaced the pins. The checks + # themselves live in tests/stamps/run.sh so they can be run and tested + # outside CI (`make test-stamps`, tests/stamps/run_test.sh) rather than + # existing only as inline YAML; it reads the expected format out of the + # datastream, so the guard cannot drift from the OVAL it guards. # - # The jre image is used because it carries BOTH stamps, so a single - # export covers the whole rule. wolfi-base has no Java and would leave - # the truststore criterion unguarded. - - workdir=$(mktemp -d) - crane export "${STEPS_IMAGE_OUTPUTS_STAMP_FULL_REF}" - | \ - tar -C "${workdir}" -x \ - etc/ssl/certs/ca-certificates.crt \ - etc/ssl/certs/.ca-certificates.crt.sha256 \ - etc/ssl/certs/java/cacerts \ - etc/ssl/certs/java/.cacerts.sha256 - - # Both stamps must be present and correct. Checking each separately - # (rather than one combined sha256sum -c) keeps the error message - # specific about which trust store drifted. - for pair in "etc/ssl/certs:.ca-certificates.crt.sha256" "etc/ssl/certs/java:.cacerts.sha256"; do - dir="${pair%%:*}" - stamp="${pair#*:}" - if [ ! -f "${workdir}/${dir}/${stamp}" ]; then - echo "::error::${dir}/${stamp} missing from ${STAMP_IMAGE_REF}; CertificateAudit will fail on clean images" - exit 1 - fi - if ! (cd "${workdir}/${dir}" && sha256sum -c "${stamp}"); then - echo "::error::${dir} contents do not match ${stamp} in ${STAMP_IMAGE_REF}; CertificateAudit will fail on clean images" - exit 1 - fi - done - - SHA=$(sha256sum "${workdir}/etc/ssl/certs/ca-certificates.crt" | cut -d' ' -f1) - JAVA_SHA=$(sha256sum "${workdir}/etc/ssl/certs/java/cacerts" | cut -d' ' -f1) + # The jre image is used because it carries BOTH sidecars the rule + # reads, so a single export covers them. wolfi-base has no Java and + # would leave the truststore criterion unguarded. + + digests="${RUNNER_TEMP}/stamp-digests" + : > "${digests}" + STAMP_DIGEST_FILE="${digests}" \ + tests/stamps/run.sh "${STEPS_IMAGE_OUTPUTS_STAMP_FULL_REF}" + + # Reported for the summary below; the guard already proved each of + # these equals the digest recorded in the sidecar beside it. + SHA=$(awk '$1=="etc/ssl/certs/ca-certificates.crt"{print $2}' "${digests}") + JAVA_SHA=$(awk '$1=="etc/ssl/certs/java/cacerts"{print $2}' "${digests}") echo "sha=${SHA}" >> "$GITHUB_OUTPUT" cat >> "$GITHUB_STEP_SUMMARY" < ", for a caller that wants to +# report the digests without exporting the image again +# +# Exits non-zero, naming the trust store, on the first violation. + +set -euo pipefail + +REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)" +DATASTREAM="${DATASTREAM:-${REPO_ROOT}/gpos/xml/scap/ssg/content/ssg-chainguard-gpos-ds.xml}" + +# jre carries both of the sidecars the rule reads on a public image — the +# system CA bundle's and the Java truststore's — so one export covers both. +# wolfi-base has no Java and would leave the truststore criterion unguarded. +DEFAULT_IMAGES="${STAMP_IMAGES:-cgr.dev/chainguard/jre:latest}" + +# Sidecars the OVAL reads, and whether the rule tolerates one being absent. +# +# |||| +# +# required=yes the criteria demand the sidecar (tst:4, tst:6), so a trust +# store present without one is a failure. +# required=no the criteria accept its absence and fall back to the system +# sidecar (tst:13 + tst:9), so only a present-but-wrong sidecar +# is a failure. +# +# /kaniko is listed for completeness but is not reachable on a public image; +# see the coverage note printed at the end of a run. +SIDECARS=( + "oval:org.CABundleHash:obj:4|etc/ssl/certs|.ca-certificates.crt.sha256|ca-certificates.crt|yes" + "oval:org.CABundleHash:obj:6|etc/ssl/certs/java|.cacerts.sha256|cacerts|yes" + "oval:org.CABundleHash:obj:10|kaniko/ssl/certs|.ca-certificates.crt.sha256|ca-certificates.crt|no" +) + +die() { + # ::error:: is picked up as an annotation under Actions and is harmless + # otherwise, so the same script serves CI and a local run. + echo "::error::$*" >&2 + exit 1 +} + +note() { echo " $*"; } + +# Print the the datastream gives an OVAL object, so this guard +# asserts the regex the scanner will actually apply rather than a copy of it. +oval_pattern() { + local obj_id="$1" + # The parsed input is this repository's own datastream, not untrusted data, + # so the stdlib parser is used rather than taking on a defusedxml dependency. + python3 - "$DATASTREAM" "$obj_id" <<'PY' +import sys, xml.etree.ElementTree as ET +ds, obj_id = sys.argv[1], sys.argv[2] +local = lambda e: e.tag.split('}')[-1] +for e in ET.parse(ds).iter(): + if local(e) == 'textfilecontent54_object' and e.get('id') == obj_id: + for c in e: + if local(c) == 'pattern': + print((c.text or '').strip()) + sys.exit(0) +sys.exit(f"no pattern for {obj_id} in {ds}") +PY +} + +check_image() { + local ref="$1" workdir checked=0 + workdir="$(mktemp -d)" + # shellcheck disable=SC2064 # expand workdir now, not at trap time + trap "rm -rf '${workdir}'" RETURN + + echo "== ${ref}" + + # Extract the containing directories rather than naming each file. `tar -x + # ` exits 2 when a member is absent from the archive, which under + # `set -e` would kill this function before the per-file checks below could + # say *which* sidecar was missing — the exact diagnosis this guard exists to + # produce. Directories that the image does not have are simply not extracted. + local dirs=() seen="" + local entry dir + for entry in "${SIDECARS[@]}"; do + dir="$(echo "${entry}" | cut -d'|' -f2)" + case " ${seen} " in *" ${dir} "*) continue ;; esac + seen="${seen} ${dir}" + dirs+=("${dir}") + done + + # A missing directory is not an error here (no Java, no /kaniko), so ask tar + # for each separately and let absence be silent. `|| true` is safe because + # every file is then explicitly accounted for below. + local tarball="${workdir}/image.tar" + crane export "${ref}" - > "${tarball}" \ + || die "could not export ${ref}" + for dir in "${dirs[@]}"; do + tar -C "${workdir}" -xf "${tarball}" "${dir}" 2>/dev/null || true + done + rm -f "${tarball}" + + for entry in "${SIDECARS[@]}"; do + local obj_id sidecar store required pattern + obj_id="$(echo "${entry}" | cut -d'|' -f1)" + dir="$(echo "${entry}" | cut -d'|' -f2)" + sidecar="$(echo "${entry}" | cut -d'|' -f3)" + store="$(echo "${entry}" | cut -d'|' -f4)" + required="$(echo "${entry}" | cut -d'|' -f5)" + + # No trust store at this path means the rule has nothing to verify here. + if [ ! -f "${workdir}/${dir}/${store}" ]; then + note "${dir}/${store}: absent, nothing to verify" + continue + fi + + if [ ! -f "${workdir}/${dir}/${sidecar}" ]; then + if [ "${required}" = "yes" ]; then + die "${dir}/${sidecar} missing from ${ref}, but ${dir}/${store} is present; CertificateAudit will fail on clean images" + fi + note "${dir}/${sidecar}: absent, permitted (rule falls back to the system sidecar)" + continue + fi + + # (3) format: the regex the OVAL will apply must match this sidecar. + pattern="$(oval_pattern "${obj_id}")" + if ! grep -Pq -- "${pattern}" "${workdir}/${dir}/${sidecar}"; then + die "${dir}/${sidecar} in ${ref} does not match the pattern ${obj_id} applies (${pattern}); CertificateAudit will fail on clean images" + fi + + # (2) contents: the digest must describe the file it names. + if ! (cd "${workdir}/${dir}" && sha256sum -c "${sidecar}" >/dev/null); then + die "${dir}/${store} in ${ref} does not match the digest in ${sidecar}; CertificateAudit will fail on clean images" + fi + + note "${dir}/${store}: matches ${sidecar}, and ${sidecar} matches ${obj_id}'s pattern" + if [ -n "${STAMP_DIGEST_FILE:-}" ]; then + printf '%s %s\n' "${dir}/${store}" \ + "$(sha256sum "${workdir}/${dir}/${store}" | cut -d' ' -f1)" \ + >> "${STAMP_DIGEST_FILE}" + fi + checked=$((checked + 1)) + done + + [ "${checked}" -gt 0 ] \ + || die "no trust store was verified in ${ref}; the guard checked nothing and would pass vacuously" +} + +main() { + local images=("$@") + [ "${#images[@]}" -gt 0 ] || read -r -a images <<<"${DEFAULT_IMAGES}" + + for ref in "${images[@]}"; do + check_image "${ref}" + done + + # Say what was not covered, so a green run is not read as "the whole rule's + # premise is guarded". + cat <<'EOF' + +Not covered by this guard: + - /kaniko/ssl/certs — only present on cgr.dev/chainguard-private/kaniko, + which needs registry credentials this workflow does not have. + - var/lib/ecs/deps/execute-command/certs/tls-ca-bundle.pem — the image build + writes a sidecar for it, but CertificateAudit does not read it, so there is + no premise to guard yet. +EOF +} + +main "$@" diff --git a/tests/stamps/run_test.sh b/tests/stamps/run_test.sh new file mode 100755 index 0000000..2fd7c54 --- /dev/null +++ b/tests/stamps/run_test.sh @@ -0,0 +1,144 @@ +#!/usr/bin/env bash +# +# SPDX-License-Identifier: Apache-2.0 +# Copyright (c) 2026 Chainguard +# +# Tests tests/stamps/run.sh against synthetic images. +# +# The guard's whole value is in its failure paths, and those are the paths a +# green CI run never exercises: the daily run only ever sees a healthy image. +# Two of them were previously unreachable or absent altogether — +# +# - a missing sidecar aborted on `tar`'s "Not found in archive" before the +# diagnostic naming which trust store could run, and +# - a sidecar whose digest is right but whose filename spelling the OVAL's +# regex rejects passed a `sha256sum -c`-only guard while failing the rule. +# +# so each is pinned by a case below. +# +# `crane` is stubbed with a script that streams a tar of a prepared directory, +# which keeps the test hermetic: no registry, no network, and mutations that +# would be awkward to publish as real images are just files on disk. + +set -euo pipefail + +HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +# Overridable so the suite can be pointed at a deliberately-broken copy of the +# guard to confirm these cases actually fail when the behaviour regresses. +GUARD="${STAMP_GUARD:-${HERE}/run.sh}" + +WORK="$(mktemp -d)" +trap 'rm -rf "${WORK}"' EXIT + +VARIANTS="${WORK}/variants" +mkdir -p "${WORK}/bin" "${VARIANTS}" + +cat > "${WORK}/bin/crane" <<'EOS' +#!/usr/bin/env bash +# Stub. `crane export -` streams a tar of the prepared directory, +# shaped like real crane output (no leading "./"). +[ "$1" = "export" ] || { echo "stub crane: unexpected args: $*" >&2; exit 64; } +d="${STAMP_TEST_VARIANTS}/$2" +[ -d "${d}" ] || { echo "stub crane: no such variant: $2" >&2; exit 1; } +cd "${d}" && exec tar -cf - $(ls -A) +EOS +chmod +x "${WORK}/bin/crane" +export PATH="${WORK}/bin:${PATH}" +export STAMP_TEST_VARIANTS="${VARIANTS}" + +# A pristine image: two trust stores, each with a sidecar that describes it. +base="${WORK}/base" +mkdir -p "${base}/etc/ssl/certs/java" "${base}/kaniko/ssl/certs" +echo "bundle content" > "${base}/etc/ssl/certs/ca-certificates.crt" +echo "truststore content" > "${base}/etc/ssl/certs/java/cacerts" +sidecar_for() { + printf '%s %s\n' "$(sha256sum "$1" | cut -d' ' -f1)" "$(basename "$1")" +} +sidecar_for "${base}/etc/ssl/certs/ca-certificates.crt" \ + > "${base}/etc/ssl/certs/.ca-certificates.crt.sha256" +sidecar_for "${base}/etc/ssl/certs/java/cacerts" \ + > "${base}/etc/ssl/certs/java/.cacerts.sha256" + +variant() { rm -rf "${VARIANTS:?}/$1"; cp -r "${base}" "${VARIANTS}/$1"; echo "${VARIANTS}/$1"; } + +v="$(variant clean)" + +v="$(variant no_java)"; rm -rf "${v}/etc/ssl/certs/java" + +v="$(variant kaniko_without_sidecar)" +cp "${base}/etc/ssl/certs/ca-certificates.crt" "${v}/kaniko/ssl/certs/ca-certificates.crt" + +v="$(variant missing_system_sidecar)"; rm "${v}/etc/ssl/certs/.ca-certificates.crt.sha256" + +v="$(variant missing_java_sidecar)"; rm "${v}/etc/ssl/certs/java/.cacerts.sha256" + +v="$(variant wrong_digest)" +printf '%064d ca-certificates.crt\n' 0 > "${v}/etc/ssl/certs/.ca-certificates.crt.sha256" + +# Digest is correct; only the filename field's spelling differs. `sha256sum -c` +# accepts this, the OVAL's regex does not. +v="$(variant absolute_path_sidecar)" +( cd "${v}/etc/ssl/certs" \ + && printf '%s /etc/ssl/certs/ca-certificates.crt\n' \ + "$(sha256sum ca-certificates.crt | cut -d' ' -f1)" \ + > .ca-certificates.crt.sha256 ) + +v="$(variant kaniko_sidecar_disagrees)" +cp "${base}/etc/ssl/certs/ca-certificates.crt" "${v}/kaniko/ssl/certs/ca-certificates.crt" +printf '%064d ca-certificates.crt\n' 0 > "${v}/kaniko/ssl/certs/.ca-certificates.crt.sha256" + +# name | variant | want_exit | substring the output must contain +CASES=( + "pristine image passes|clean|0|matches .ca-certificates.crt.sha256" + "image without Java skips the truststore|no_java|0|etc/ssl/certs/java/cacerts: absent" + "kaniko copy without a sidecar is permitted|kaniko_without_sidecar|0|absent, permitted" + "missing system sidecar is named, not a tar error|missing_system_sidecar|1|etc/ssl/certs/.ca-certificates.crt.sha256 missing" + "missing Java sidecar is named, not a tar error|missing_java_sidecar|1|etc/ssl/certs/java/.cacerts.sha256 missing" + "sidecar digest that disagrees fails|wrong_digest|1|does not match the digest" + "sidecar the OVAL regex rejects fails despite sha256sum -c passing|absolute_path_sidecar|1|does not match the pattern" + "kaniko sidecar that disagrees fails|kaniko_sidecar_disagrees|1|does not match the digest" +) + +failed=0 +for case in "${CASES[@]}"; do + IFS='|' read -r name ref want_exit want_substr <<<"${case}" + + set +e + out="$("${GUARD}" "${ref}" 2>&1)" + got_exit=$? + set -e + + if [ "${got_exit}" -ne "${want_exit}" ]; then + printf 'FAIL %s\n variant=%s want exit %s, got %s\n output:\n%s\n' \ + "${name}" "${ref}" "${want_exit}" "${got_exit}" "${out}" + failed=1 + continue + fi + if ! printf '%s' "${out}" | grep -qF -- "${want_substr}"; then + printf 'FAIL %s\n variant=%s exit %s as expected, but output lacked %q\n output:\n%s\n' \ + "${name}" "${ref}" "${got_exit}" "${want_substr}" "${out}" + failed=1 + continue + fi + printf 'ok %s\n' "${name}" +done + +# The guard must refuse to report success when it verified nothing at all, +# rather than treating an image with no trust stores as compliant. +v="$(variant empty)"; rm -rf "${v:?}/etc" "${v:?}/kaniko"; mkdir -p "${v}/usr" +set +e +out="$("${GUARD}" empty 2>&1)"; got_exit=$? +set -e +if [ "${got_exit}" -eq 0 ] || ! printf '%s' "${out}" | grep -qF "pass vacuously"; then + printf 'FAIL guard rejects an image where nothing was verified\n want non-zero and a vacuous-pass error, got exit %s:\n%s\n' \ + "${got_exit}" "${out}" + failed=1 +else + printf 'ok guard rejects an image where nothing was verified\n' +fi + +if [ "${failed}" -ne 0 ]; then + echo "FAILED" >&2 + exit 1 +fi +echo "all stamp-guard cases passed" From 6b8a3d7af91d2e001621da55d8cc1d80a95522a2 Mon Sep 17 00:00:00 2001 From: Steve Beattie Date: Thu, 20 Aug 2026 12:53:43 -0700 Subject: [PATCH 2/3] Hold a sidecar-less bundle copy to the system sidecar it falls back to MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Running the guard against cgr.dev/chainguard-private/kaniko showed it reporting the /kaniko copy as "absent, permitted" and checking nothing further. That is wrong. The criteria permit a copy without a sidecar only by holding it to the system sidecar instead (tst:13 + tst:9), so the fallback carries a premise of its own, and the guard was waving it through unverified — a copy that diverged from the system bundle would pass here and fail the rule, which is exactly the miss this guard exists to prevent. Verify it: with no sidecar beside the copy, compare the copy against the digest in the system sidecar and fail, naming both digests, when they differ. Refuse the case where there is no system sidecar to fall back to either. Confirmed against the real image — kaniko's copy is byte-identical to its system bundle, so it passes for the right reason now rather than by not being looked at. A synthetic diverged copy is covered by a new case, which fails against the previous behaviour. Also stop claiming /kaniko is uncovered when it was in fact inspected. The closing note now reports what the run actually reached, so passing the private ref explicitly — which developers with access can do locally — drops the caveat instead of printing it misleadingly. Co-Authored-By: Claude Opus 5 --- tests/stamps/run.sh | 58 +++++++++++++++++++++++++++++++--------- tests/stamps/run_test.sh | 10 ++++++- 2 files changed, 55 insertions(+), 13 deletions(-) diff --git a/tests/stamps/run.sh b/tests/stamps/run.sh index 571f168..e0d91fa 100755 --- a/tests/stamps/run.sh +++ b/tests/stamps/run.sh @@ -52,14 +52,19 @@ DEFAULT_IMAGES="${STAMP_IMAGES:-cgr.dev/chainguard/jre:latest}" # sidecar (tst:13 + tst:9), so only a present-but-wrong sidecar # is a failure. # -# /kaniko is listed for completeness but is not reachable on a public image; -# see the coverage note printed at the end of a run. +# /kaniko is carried only by cgr.dev/chainguard-private/kaniko, so it is not +# reached by the default public image. Pass that ref explicitly to cover it +# wherever credentials for it exist; a run reports what it did not reach. SIDECARS=( "oval:org.CABundleHash:obj:4|etc/ssl/certs|.ca-certificates.crt.sha256|ca-certificates.crt|yes" "oval:org.CABundleHash:obj:6|etc/ssl/certs/java|.cacerts.sha256|cacerts|yes" "oval:org.CABundleHash:obj:10|kaniko/ssl/certs|.ca-certificates.crt.sha256|ca-certificates.crt|no" ) +# Directories this run actually inspected, so the closing coverage note can +# report what was left out rather than a fixed list. +INSPECTED_DIRS="" + die() { # ::error:: is picked up as an annotation under Actions and is harmless # otherwise, so the same script serves CI and a local run. @@ -122,6 +127,14 @@ check_image() { done rm -f "${tarball}" + # The digest the criteria fall back to for a bundle copy that ships no + # sidecar of its own. Read once, before the loop, so the fallback check does + # not depend on the order of SIDECARS. + local system_digest="" + if [ -f "${workdir}/etc/ssl/certs/.ca-certificates.crt.sha256" ]; then + system_digest="$(cut -d' ' -f1 < "${workdir}/etc/ssl/certs/.ca-certificates.crt.sha256")" + fi + for entry in "${SIDECARS[@]}"; do local obj_id sidecar store required pattern obj_id="$(echo "${entry}" | cut -d'|' -f1)" @@ -140,7 +153,23 @@ check_image() { if [ "${required}" = "yes" ]; then die "${dir}/${sidecar} missing from ${ref}, but ${dir}/${store} is present; CertificateAudit will fail on clean images" fi - note "${dir}/${sidecar}: absent, permitted (rule falls back to the system sidecar)" + # Absence is permitted, but not unconditionally: the criteria fall back + # to holding this copy to the *system* sidecar (tst:13 + tst:9), so the + # fallback has a premise of its own and it has to be checked here too. + # Permitting the branch without checking it would let a diverged copy + # pass this guard and fail the rule, which is the miss the guard exists + # to prevent. + if [ -z "${system_digest}" ]; then + die "${dir}/${store} in ${ref} has no sidecar and there is no system sidecar to fall back to; CertificateAudit will fail on clean images" + fi + local copy_digest + copy_digest="$(sha256sum "${workdir}/${dir}/${store}" | cut -d' ' -f1)" + if [ "${copy_digest}" != "${system_digest}" ]; then + die "${dir}/${store} in ${ref} ships no sidecar and does not match the system sidecar it falls back to (${copy_digest} vs ${system_digest}); CertificateAudit will fail on clean images" + fi + note "${dir}/${store}: no sidecar of its own, and matches the system sidecar it falls back to" + INSPECTED_DIRS="${INSPECTED_DIRS} ${dir}" + checked=$((checked + 1)) continue fi @@ -156,6 +185,7 @@ check_image() { fi note "${dir}/${store}: matches ${sidecar}, and ${sidecar} matches ${obj_id}'s pattern" + INSPECTED_DIRS="${INSPECTED_DIRS} ${dir}" if [ -n "${STAMP_DIGEST_FILE:-}" ]; then printf '%s %s\n' "${dir}/${store}" \ "$(sha256sum "${workdir}/${dir}/${store}" | cut -d' ' -f1)" \ @@ -178,15 +208,19 @@ main() { # Say what was not covered, so a green run is not read as "the whole rule's # premise is guarded". - cat <<'EOF' - -Not covered by this guard: - - /kaniko/ssl/certs — only present on cgr.dev/chainguard-private/kaniko, - which needs registry credentials this workflow does not have. - - var/lib/ecs/deps/execute-command/certs/tls-ca-bundle.pem — the image build - writes a sidecar for it, but CertificateAudit does not read it, so there is - no premise to guard yet. -EOF + echo + echo "Not covered by this run:" + case " ${INSPECTED_DIRS} " in + *" kaniko/ssl/certs "*) ;; + *) + echo " - /kaniko/ssl/certs — no inspected image carried it. It exists on" + echo " cgr.dev/chainguard-private/kaniko, which needs registry credentials;" + echo " pass that ref explicitly to cover it where they are available." + ;; + esac + echo " - var/lib/ecs/deps/execute-command/certs/tls-ca-bundle.pem — the image" + echo " build writes a sidecar for it, but CertificateAudit does not read it," + echo " so there is no premise to guard yet." } main "$@" diff --git a/tests/stamps/run_test.sh b/tests/stamps/run_test.sh index 2fd7c54..789ca72 100755 --- a/tests/stamps/run_test.sh +++ b/tests/stamps/run_test.sh @@ -87,11 +87,19 @@ v="$(variant kaniko_sidecar_disagrees)" cp "${base}/etc/ssl/certs/ca-certificates.crt" "${v}/kaniko/ssl/certs/ca-certificates.crt" printf '%064d ca-certificates.crt\n' 0 > "${v}/kaniko/ssl/certs/.ca-certificates.crt.sha256" +# A copy with no sidecar of its own is held to the system sidecar instead +# (tst:13 + tst:9), so "no sidecar" is permitted only while the copy still +# matches. A diverged copy has to fail: permitting the branch without checking +# its premise would pass an image the rule rejects. +v="$(variant kaniko_copy_diverges)" +echo "a different bundle" > "${v}/kaniko/ssl/certs/ca-certificates.crt" + # name | variant | want_exit | substring the output must contain CASES=( "pristine image passes|clean|0|matches .ca-certificates.crt.sha256" "image without Java skips the truststore|no_java|0|etc/ssl/certs/java/cacerts: absent" - "kaniko copy without a sidecar is permitted|kaniko_without_sidecar|0|absent, permitted" + "kaniko copy without a sidecar passes while it matches the system one|kaniko_without_sidecar|0|matches the system sidecar it falls back to" + "kaniko copy without a sidecar that diverges from the system one fails|kaniko_copy_diverges|1|does not match the system sidecar it falls back to" "missing system sidecar is named, not a tar error|missing_system_sidecar|1|etc/ssl/certs/.ca-certificates.crt.sha256 missing" "missing Java sidecar is named, not a tar error|missing_java_sidecar|1|etc/ssl/certs/java/.cacerts.sha256 missing" "sidecar digest that disagrees fails|wrong_digest|1|does not match the digest" From 207e834dea73c03b9413c6cc141f45a06ead0464 Mon Sep 17 00:00:00 2001 From: Steve Beattie Date: Fri, 21 Aug 2026 14:27:17 -0700 Subject: [PATCH 3/3] Document covering the /kaniko criteria locally MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The /kaniko bundle-copy criteria are only reachable on a private image, so no automated run covers them. That is a deliberate deferral rather than a decision not to check: automating it needs two additions to the daily workflow's trust surface, not one — a credential for the private registry, and a second accepted signer identity, since that image is signed by chainguard-dev/stereo's release workflow rather than the chainguard-images/images identity the workflow requires. The copy is currently byte-identical to its system bundle, so the drift is remote. What is available now is covering it on demand, which anyone with access to the image can do by naming it. Document that, in the guard's own docs and in the README's testing section alongside the other tiers, together with the credential requirement it needs. That requirement is worth spelling out because the failure is confusing: cgr.dev is served by a credential helper that wants a token issued for the cgr.dev audience, and a plain `chainctl auth login` does not produce one. `chainctl auth status` then reports Valid: True while the pull fails with "No matching credentials were found", which reads as a broken guard rather than a missing audience. The fix, `chainctl auth login --audience=cgr.dev`, is now named both in the docs and in the guard's own error message. Verified end to end: the documented invocation passes against the real private image, with the /kaniko copy checked against the system sidecar it falls back to, and the /kaniko caveat correctly dropped from the closing coverage note. Co-Authored-By: Claude Opus 5 --- README.md | 32 ++++++++++++++++++++ docs/certificate-audit.md | 62 +++++++++++++++++++++++++++++++++++++++ tests/stamps/run.sh | 2 +- 3 files changed, 95 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index 3a20f4c..0a47c18 100644 --- a/README.md +++ b/README.md @@ -165,3 +165,35 @@ Adding a fixture: - Create `tests/e2e/fixtures//expected.txt` listing each rule ID you want to assert on, one per line, as `=`. + +### Trust-store sidecar guard + +Both tiers above scan images this repository builds. `CertificateAudit` +additionally depends on something no fixture can assert: that *real* +images still ship the trust-store checksum sidecars it reads, in the +format its regexes expect. `make test-stamps` checks that premise +against a live image, reading the expected patterns out of the +datastream so the guard cannot drift from the definition it guards. The +daily `.github/workflows/update-ca-cert.yaml` run invokes the same +script, so an upstream change fails there rather than surfacing later as +an unexplained failure on clean images. + +```bash +make test-stamps # default: cgr.dev/chainguard/jre:latest +make test-stamps-selftest # the guard's own failure paths; hermetic +``` + +The self-test stubs `crane` and builds its images on disk, so it needs +no registry or network, and it runs on PRs via the offline workflow. It +covers the failure paths specifically because a green daily run only +ever sees a healthy image. + +Pass image references to widen coverage — `crane` uses the same registry +credentials as `docker`. The `/kaniko` bundle-copy criteria are only +reachable on a private image, so the default run does not cover them and +says so; see +[docs/certificate-audit.md](./docs/certificate-audit.md#covering-the-kaniko-criteria). + +```bash +make test-stamps STAMP_IMAGES="cgr.dev/chainguard/jre:latest cgr.dev/chainguard-private/kaniko:latest" +``` diff --git a/docs/certificate-audit.md b/docs/certificate-audit.md index bef384d..31a5cdd 100644 --- a/docs/certificate-audit.md +++ b/docs/certificate-audit.md @@ -280,6 +280,64 @@ Two cautions when editing that matrix: variables, so a fixture blob can exist with no row referencing it and the suite still passes. Check the subtest count, not just the diff. +### Guarding the premise + +Those fixtures test the rule against images this repository builds. They cannot +tell you that a *real* image still ships sidecars in the shape the rule expects +— which is the assumption the whole rule now rests on, and which upstream can +change without warning. `tests/stamps/run.sh` covers that: for each trust store +in an image it checks the sidecar exists, agrees with the file it names, and +matches the regex the OVAL will apply. The pattern is read out of the datastream +rather than copied, so the guard cannot drift from the definition it guards. + + make test-stamps # default: cgr.dev/chainguard/jre:latest + make test-stamps-selftest # failure paths, hermetic, no registry + +`jre` is the default because it is a public image carrying both the system and +Java sidecars. The daily `update-ca-cert` workflow runs the same script, so a +sidecar that disappears or changes format upstream fails there rather than +surfacing later as a mysterious `CertificateAudit` failure on clean images. + +`run_test.sh` is where the failure paths live, with `crane` stubbed so the cases +need no registry or network. It matters because a green daily run only ever sees +a healthy image, so nothing else exercises the diagnostics. + +#### Covering the /kaniko criteria + +The `/kaniko` bundle copy exists only on `cgr.dev/chainguard-private/kaniko`, so +the default run does not reach it and says so. Anyone with access to that image +can cover it locally by naming it. + +`crane` reads the same credentials as `docker`, and `cgr.dev` is served by the +`cgr` credential helper, which needs a token issued for the `cgr.dev` audience +specifically: + + chainctl auth login --audience=cgr.dev + +A plain `chainctl auth login` is not enough — that token is scoped to the +console API, so `chainctl auth status` reports `Valid: True` while the pull +still fails with `No matching credentials were found for "cgr.dev"`. The +credential helper itself is usually already wired up (`chainctl auth +configure-docker` will say so); the audience is the part that goes missing. + + tests/stamps/run.sh cgr.dev/chainguard/jre:latest \ + cgr.dev/chainguard-private/kaniko:latest + + # or, equivalently + make test-stamps STAMP_IMAGES="cgr.dev/chainguard/jre:latest cgr.dev/chainguard-private/kaniko:latest" + +A run reports what it did not reach, so passing that ref drops the `/kaniko` +caveat from the closing note instead of printing it. + +This is deliberately not automated. Doing so would need two additions to the +workflow's trust surface, not one: a credential for the private registry, and a +second accepted signer identity, because that image is signed by +`chainguard-dev/stereo/.github/workflows/release-containers.yaml` rather than +the `chainguard-images/images/*` identity the workflow requires. The copy is +currently byte-identical to its system bundle, so the drift being guarded +against is remote; the trade was judged not worth it for now. Revisit if the +`/kaniko` copy ever starts diverging, or gains a sidecar of its own. + ## Known gaps - apko also stamps `var/lib/ecs/deps/execute-command/certs/tls-ca-bundle.pem` @@ -293,3 +351,7 @@ Two cautions when editing that matrix: containing two matching lines silently uses the first. - Deleting `/etc/ssl/certs/java/cacerts` outright satisfies the Java `OR` via `tst:5`. +- No automated run guards the `/kaniko` criteria against a real image; the + daily workflow inspects only public images. It is coverable on demand — see + [Covering the /kaniko criteria](#covering-the-kaniko-criteria) — and deferred + rather than declined. diff --git a/tests/stamps/run.sh b/tests/stamps/run.sh index e0d91fa..34e3647 100755 --- a/tests/stamps/run.sh +++ b/tests/stamps/run.sh @@ -121,7 +121,7 @@ check_image() { # every file is then explicitly accounted for below. local tarball="${workdir}/image.tar" crane export "${ref}" - > "${tarball}" \ - || die "could not export ${ref}" + || die "could not export ${ref} (for a private ref, check registry credentials — cgr.dev needs a token issued for its own audience: 'chainctl auth login --audience=cgr.dev')" for dir in "${dirs[@]}"; do tar -C "${workdir}" -xf "${tarball}" "${dir}" 2>/dev/null || true done