Make the trust-store sidecar guard reachable, format-aware and testable - #160
Open
stevebeattie wants to merge 3 commits into
Open
Make the trust-store sidecar guard reachable, format-aware and testable#160stevebeattie wants to merge 3 commits into
stevebeattie wants to merge 3 commits into
Conversation
The daily guard that checks CertificateAudit's premise had two holes, both in code paths a healthy image never reaches, so neither could show up in a green run. First, the missing-sidecar diagnostic was unreachable. Extraction named each file individually, and `tar -x <member>` exits 2 when a member is absent from the archive, so under `set -euo pipefail` the step died on tar's "Not found in archive" before reaching the branch that says which trust store lost its sidecar — the exact diagnosis the guard exists to produce. Extraction now asks for the containing directories, and absence is reported by the per-file checks. Second, the guard checked the digest but not the format. The OVAL matches each sidecar with a regex; `sha256sum -c` is far more permissive. A sidecar naming an absolute path verifies fine under `sha256sum -c` and fails the rule, so a format change upstream would pass this guard and start failing CertificateAudit on clean images — precisely what the guard is meant to catch first. The patterns are now read out of the datastream and applied with `grep -P`, so the guard cannot drift from the OVAL it guards. The checks move from inline YAML to tests/stamps/run.sh, which makes them runnable outside CI (`make test-stamps`) and lets the failure paths be tested at all. tests/stamps/run_test.sh covers them against synthetic images with `crane` stubbed — hermetic, no registry, no network — and runs on PRs via the offline workflow. Confirmed the cases fail when each bug is reintroduced: removing the pattern assertion fails exactly the absolute-path case. The guard also now refuses to report success if it verified nothing, and prints what it does not cover, so a green run is not read as covering the whole rule. Two things remain uncovered and are named in that output: /kaniko/ssl/certs, which exists only on a private image this workflow has no credentials for, and the ECS bundle, which the image build stamps but CertificateAudit does not read. Verified: shellcheck clean; zizmor no findings; actionlint issue count unchanged from main; `make test-stamps` passes against the live jre image and `make test-stamps-selftest` passes all nine cases. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Running the guard against cgr.dev/chainguard-private/kaniko showed it reporting the /kaniko copy as "absent, permitted" and checking nothing further. That is wrong. The criteria permit a copy without a sidecar only by holding it to the system sidecar instead (tst:13 + tst:9), so the fallback carries a premise of its own, and the guard was waving it through unverified — a copy that diverged from the system bundle would pass here and fail the rule, which is exactly the miss this guard exists to prevent. Verify it: with no sidecar beside the copy, compare the copy against the digest in the system sidecar and fail, naming both digests, when they differ. Refuse the case where there is no system sidecar to fall back to either. Confirmed against the real image — kaniko's copy is byte-identical to its system bundle, so it passes for the right reason now rather than by not being looked at. A synthetic diverged copy is covered by a new case, which fails against the previous behaviour. Also stop claiming /kaniko is uncovered when it was in fact inspected. The closing note now reports what the run actually reached, so passing the private ref explicitly — which developers with access can do locally — drops the caveat instead of printing it misleadingly. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The /kaniko bundle-copy criteria are only reachable on a private image, so no automated run covers them. That is a deliberate deferral rather than a decision not to check: automating it needs two additions to the daily workflow's trust surface, not one — a credential for the private registry, and a second accepted signer identity, since that image is signed by chainguard-dev/stereo's release workflow rather than the chainguard-images/images identity the workflow requires. The copy is currently byte-identical to its system bundle, so the drift is remote. What is available now is covering it on demand, which anyone with access to the image can do by naming it. Document that, in the guard's own docs and in the README's testing section alongside the other tiers, together with the credential requirement it needs. That requirement is worth spelling out because the failure is confusing: cgr.dev is served by a credential helper that wants a token issued for the cgr.dev audience, and a plain `chainctl auth login` does not produce one. `chainctl auth status` then reports Valid: True while the pull fails with "No matching credentials were found", which reads as a broken guard rather than a missing audience. The fix, `chainctl auth login --audience=cgr.dev`, is now named both in the docs and in the guard's own error message. Verified end to end: the documented invocation passes against the real private image, with the /kaniko copy checked against the system sidecar it falls back to, and the /kaniko caveat correctly dropped from the closing coverage note. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
egibs
approved these changes
Aug 22, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
CertificateAuditpins no digest — it reads each expected value from a sidecarfile the image build writes beside the trust store. The daily
update-ca-certworkflow guards that premise. This fixes three ways the guard was not actually
doing so, and makes its failure paths testable.
Follows #158. No OVAL logic and no datastream changes here.
The holes
All three lived in paths a healthy image never reaches, so a green daily run
could not have surfaced any of them.
1. The missing-sidecar diagnostic was unreachable. Extraction named each
file individually, and
tar -x <member>exits 2 when a member is absent, sounder
set -euo pipefailthe step died on tar's "Not found in archive" beforereaching the branch that says which trust store lost its sidecar — the exact
diagnosis the guard exists to produce. It now extracts the containing
directories and lets the per-file checks report absence.
2. It checked the digest but not the format. The OVAL matches each sidecar
with a regex;
sha256sum -cis far more permissive. A sidecar naming anabsolute path verifies fine under
sha256sum -cand fails the rule — so aformat change upstream would pass this guard and start failing
CertificateAuditon clean images, which is precisely what the guard is meantto catch first. The patterns are now read out of the datastream and applied with
grep -P, so the guard cannot drift from the definition it guards.3. It permitted the
/kanikofallback without checking the fallback's ownpremise. Found by running the guard against the real private kaniko image: it
reported the absent
/kanikosidecar as "permitted" and checked nothingfurther. But the criteria permit a copy without its own sidecar only by holding
it to the system sidecar instead (
tst:13+tst:9). A copy that divergedwould have passed here and failed the rule. Now verified, naming both digests on
mismatch.
Structure
The checks move from inline YAML into
tests/stamps/run.sh, which is what makesthe failure paths testable at all, and lets them be run outside CI:
tests/stamps/run_test.shcovers ten cases against synthetic images withcranestubbed — no registry, no network — and runs on PRs via the offlineworkflow. Confirmed the cases fail when each bug is reintroduced: removing the
pattern assertion fails exactly the absolute-path case.
The guard also now refuses to report success if it verified nothing, and prints
what it did not reach, so a green run is not read as covering the whole rule.
/kaniko coverage is on-demand, deliberately
The
/kanikocriteria are only reachable oncgr.dev/chainguard-private/kaniko.Automating them would need two additions to this workflow's trust surface, not
one: a credential for the private registry, and a second accepted signer
identity, since that image is signed by
chainguard-dev/stereo/.github/workflows/release-containers.yamlrather than thechainguard-images/images/*identity the workflow requires. The copy iscurrently byte-identical to its system bundle, so the drift guarded against is
remote, and automating it is deferred rather than declined — the reasoning is
recorded in
docs/certificate-audit.mdso it can be revisited on evidence.Anyone with access can cover it now:
make test-stamps STAMP_IMAGES="cgr.dev/chainguard/jre:latest cgr.dev/chainguard-private/kaniko:latest"That needs a
cgr.dev-audience token (chainctl auth login --audience=cgr.dev).A plain
chainctl auth loginis not enough:chainctl auth statusthen reportsValid: Truewhile the pull fails withNo matching credentials were found,which reads as a broken guard rather than a missing audience. Documented, and
named in the guard's error message.
Verification
shellcheckclean on both scripts;zizmorno findings;actionlintissuecount unchanged from
main.make test-stamps-selftest— 10/10 cases.make test-stampspasses against the livejreimage, and the documentedinvocation passes against the real private kaniko image, with its
/kanikocopy checked against the system sidecar it falls back to.
gpos/untouched.🤖 Generated with Claude Code