Skip to content

ci(check-payload): add smoke tests#79882

Open
smith-xyz wants to merge 1 commit into
openshift:mainfrom
smith-xyz:ci/check-payload-scan-presubmits-v2
Open

ci(check-payload): add smoke tests#79882
smith-xyz wants to merge 1 commit into
openshift:mainfrom
smith-xyz:ci/check-payload-scan-presubmits-v2

Conversation

@smith-xyz
Copy link
Copy Markdown
Contributor

@smith-xyz smith-xyz commented May 29, 2026

Adds some quick testing to prevent regression issues

Summary by CodeRabbit

This PR updates OpenShift CI configuration to add quick smoke tests for the check-payload component by introducing four new payload-scan presubmit jobs in the ci-operator config for the openshift/check-payload repo.

Practical effect / scope

  • Affects OpenShift CI job definitions (ci-operator config): ci-operator/config/openshift/check-payload/openshift-check-payload-main.yaml.
  • Adds four new test jobs that run in presubmits for the check-payload repo targeting nightly release streams for OpenShift versions 4.12, 4.15, 4.20, and 5.0.

What each job does (practical terms)

  • Resolves the latest nightly release pullSpec for its stream (via releases API).
  • Derives a bounded list of release image references and extracts each image filesystem locally (using oc image extract).
  • Runs the bundled check-payload scanner (/check-payload scan local) against the unpacked payload content for that version.
  • Tracks total and failed image extractions/scans and fails the job if any image fails.
  • Uses the check-payload container image, mounts the ci-pull-credentials secret for registry auth, and has a 45m timeout.

Resource / configuration changes

  • Adds resource overrides for the four new jobs under resources['*'], configuring limits.memory: 8Gi and requests.cpu: "1", requests.memory: 4Gi to accommodate extraction/scan workload.

Purpose

  • Provides lightweight smoke tests to catch regressions in check-payload by validating the tool against the latest nightly releases across multiple active OpenShift versions.

@coderabbitai
Copy link
Copy Markdown
Contributor

coderabbitai Bot commented May 29, 2026

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Repository YAML (base), Central YAML (inherited)

Review profile: CHILL

Plan: Enterprise

Run ID: f61815b0-12fe-4c90-94c8-c5836ceafdd1

📥 Commits

Reviewing files that changed from the base of the PR and between 1afea65 and b27a7bf.

⛔ Files ignored due to path filters (1)
  • ci-operator/jobs/openshift/check-payload/openshift-check-payload-main-presubmits.yaml is excluded by !ci-operator/jobs/**
📒 Files selected for processing (1)
  • ci-operator/config/openshift/check-payload/openshift-check-payload-main.yaml
🚧 Files skipped from review as they are similar to previous changes (1)
  • ci-operator/config/openshift/check-payload/openshift-check-payload-main.yaml

Walkthrough

This PR adds four CI payload-scan jobs (4.12, 4.15, 4.20, 5.0) and per-job resource overrides. Each job resolves a nightly release pullSpec, derives image references, extracts image filesystems using mounted registry credentials, and runs /check-payload scan local; jobs use the check-payload container and a 45m timeout.

Changes

Payload Scan CI Jobs

Layer / File(s) Summary
Resource configuration for payload-scan jobs
ci-operator/config/openshift/check-payload/openshift-check-payload-main.yaml
Resource requests (cpu: "1", memory: 4Gi) and limits (memory: 8Gi) are defined for payload-scan-4-12, payload-scan-4-15, payload-scan-4-20, and payload-scan-5-0.
Payload-scan job definitions
ci-operator/config/openshift/check-payload/openshift-check-payload-main.yaml
Adds four jobs (4.12, 4.15, 4.20, 5.0) that obtain nightly release pullSpecs, list release images, extract each image filesystem with ci-pull-credentials, run /check-payload scan local for the job version, count failures, and exit non-zero on any failure; each job uses the check-payload container and a 45m0s timeout.

🎯 2 (Simple) | ⏱️ ~10 minutes

Suggested labels: lgtm, rehearsals-ack

🚥 Pre-merge checks | ✅ 15
✅ Passed checks (15 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title 'ci(check-payload): add smoke tests' accurately describes the main change: adding new CI smoke test jobs for the check-payload project.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Stable And Deterministic Test Names ✅ Passed PR modifies CI operator config (YAML), not Ginkgo Go tests. Custom check is specific to Ginkgo test names and is not applicable to this change.
Test Structure And Quality ✅ Passed The custom check requires reviewing Ginkgo test code, but this PR modifies only a CI configuration YAML file with bash scripts. No Ginkgo/Go test code is present, so the check is not applicable.
Microshift Test Compatibility ✅ Passed No Ginkgo e2e tests added. PR modifies CI operator config file with bash-based payload scan jobs, not e2e tests.
Single Node Openshift (Sno) Test Compatibility ✅ Passed PR does not add any Ginkgo e2e tests (It/Describe/Context/When patterns). It only adds bash-based CI job configurations for payload scanning. The SNO compatibility check is not applicable.
Topology-Aware Scheduling Compatibility ✅ Passed The PR modifies CI-Operator configuration for test jobs, not Kubernetes deployment manifests, operator code, or controllers. No topology-aware scheduling constraints are introduced.
Ote Binary Stdout Contract ✅ Passed This PR only modifies CI job configuration (YAML) in openshift/release. The OTE Binary Stdout Contract check applies to Go source code, which is not present in this PR.
Ipv6 And Disconnected Network Test Compatibility ✅ Passed PR adds CI/CD job definitions (bash scripts) in YAML, not Ginkgo e2e tests. Custom check applies only to Ginkgo tests (It/Describe/Context/When), which are not present here.
No-Weak-Crypto ✅ Passed No weak crypto (MD5, SHA1, DES, RC4, 3DES, Blowfish, ECB), custom crypto, or non-constant-time comparisons found in the CI configuration YAML changes.
Container-Privileges ✅ Passed No privileged settings, hostPID/Network/IPC, allowPrivilegeEscalation, SYS_ADMIN, or root configurations found in the modified YAML manifest.
No-Sensitive-Data-In-Logs ✅ Passed No sensitive data exposure detected; credentials properly mounted as Kubernetes secrets and never logged; only public image references logged.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@openshift-ci
Copy link
Copy Markdown
Contributor

openshift-ci Bot commented May 29, 2026

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: smith-xyz

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-ci openshift-ci Bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label May 29, 2026
@openshift-ci openshift-ci Bot requested review from rhmdnd and richardsonnick May 29, 2026 18:32
Copy link
Copy Markdown
Contributor

@coderabbitai coderabbitai Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick comments (1)
ci-operator/config/openshift/check-payload/openshift-check-payload-main.yaml (1)

83-97: ⚡ Quick win

Add retry/backoff for oc registry calls to reduce flake risk.

curl has retries, but oc adm release info and oc image extract do not. A transient registry/network hiccup can fail the job unnecessarily.

Suggested patch pattern
   commands: |
     set -euo pipefail
+    retry() {
+      local attempts="$1"; shift
+      local delay="$1"; shift
+      local n=1
+      until "$@"; do
+        if [[ $n -ge $attempts ]]; then
+          return 1
+        fi
+        sleep "$delay"
+        n=$((n+1))
+      done
+    }
     export REGISTRY_AUTH_FILE=/secrets/ci-pull-credentials/.dockerconfigjson
@@
-    IMAGES=$(oc adm release info "$PULLSPEC" -a "$REGISTRY_AUTH_FILE" --pullspecs -o json \
+    IMAGES=$(retry 3 5 oc adm release info "$PULLSPEC" -a "$REGISTRY_AUTH_FILE" --pullspecs -o json \
       | jq -r '[.references.spec.tags[] | select(.from.name != null) | .from.name] | .[0:15] | .[]')
@@
-      if ! oc image extract "$img" -a "$REGISTRY_AUTH_FILE" --path /:/tmp/unpacked --confirm; then
+      if ! retry 3 5 oc image extract "$img" -a "$REGISTRY_AUTH_FILE" --path /:/tmp/unpacked --confirm; then
         echo "ERROR: failed to extract $img" >&2
         failed=$((failed+1))
         continue
       fi

Also applies to: 117-131, 151-165, 185-199

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@ci-operator/config/openshift/check-payload/openshift-check-payload-main.yaml`
around lines 83 - 97, Add a retry/backoff wrapper and use it for the registry
calls (oc adm release info and oc image extract) to reduce flakes: implement a
small helper function (e.g., retry_cmd or retry_with_backoff) that accepts a
command and retries it with exponential backoff and a limited number of
attempts, then replace direct calls to oc adm release info "$PULLSPEC" -a
"$REGISTRY_AUTH_FILE" and oc image extract "$img" -a "$REGISTRY_AUTH_FILE"
--path /:/tmp/unpacked --confirm with calls to that helper; apply the same
wrapper to the other similar oc invocations referenced (the blocks around IMAGES
resolution and the image-extract loop, and the other occurrences noted) so
transient network/registry errors are retried before failing the job.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In
`@ci-operator/config/openshift/check-payload/openshift-check-payload-main.yaml`:
- Around line 83-97: Add a retry/backoff wrapper and use it for the registry
calls (oc adm release info and oc image extract) to reduce flakes: implement a
small helper function (e.g., retry_cmd or retry_with_backoff) that accepts a
command and retries it with exponential backoff and a limited number of
attempts, then replace direct calls to oc adm release info "$PULLSPEC" -a
"$REGISTRY_AUTH_FILE" and oc image extract "$img" -a "$REGISTRY_AUTH_FILE"
--path /:/tmp/unpacked --confirm with calls to that helper; apply the same
wrapper to the other similar oc invocations referenced (the blocks around IMAGES
resolution and the image-extract loop, and the other occurrences noted) so
transient network/registry errors are retried before failing the job.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository YAML (base), Central YAML (inherited)

Review profile: CHILL

Plan: Enterprise

Run ID: ced3cf1b-2098-4dd8-afb7-b907beb14111

📥 Commits

Reviewing files that changed from the base of the PR and between 7485152 and 1afea65.

⛔ Files ignored due to path filters (1)
  • ci-operator/jobs/openshift/check-payload/openshift-check-payload-main-presubmits.yaml is excluded by !ci-operator/jobs/**
📒 Files selected for processing (1)
  • ci-operator/config/openshift/check-payload/openshift-check-payload-main.yaml

@smith-xyz
Copy link
Copy Markdown
Contributor Author

/pj-rehearse pull-ci-openshift-check-payload-main-payload-scan-4-12

@openshift-merge-bot
Copy link
Copy Markdown
Contributor

@smith-xyz: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

@smith-xyz
Copy link
Copy Markdown
Contributor Author

/pj-rehearse pull-ci-openshift-check-payload-main-payload-scan-4-15 pull-ci-openshift-check-payload-main-payload-scan-4-20 pull-ci-openshift-check-payload-main-payload-scan-5-0

@openshift-merge-bot
Copy link
Copy Markdown
Contributor

@smith-xyz: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

@smith-xyz smith-xyz force-pushed the ci/check-payload-scan-presubmits-v2 branch from 1afea65 to b27a7bf Compare May 29, 2026 19:41
@openshift-merge-bot
Copy link
Copy Markdown
Contributor

[REHEARSALNOTIFIER]
@smith-xyz: the pj-rehearse plugin accommodates running rehearsal tests for the changes in this PR. Expand 'Interacting with pj-rehearse' for usage details. The following rehearsable tests have been affected by this change:

Test name Repo Type Reason
pull-ci-openshift-check-payload-main-payload-scan-4-12 openshift/check-payload presubmit Presubmit changed
pull-ci-openshift-check-payload-main-payload-scan-4-15 openshift/check-payload presubmit Presubmit changed
pull-ci-openshift-check-payload-main-payload-scan-4-20 openshift/check-payload presubmit Presubmit changed
pull-ci-openshift-check-payload-main-payload-scan-5-0 openshift/check-payload presubmit Presubmit changed
pull-ci-openshift-check-payload-main-build openshift/check-payload presubmit Ci-operator config changed
pull-ci-openshift-check-payload-main-images openshift/check-payload presubmit Ci-operator config changed
pull-ci-openshift-check-payload-main-test openshift/check-payload presubmit Ci-operator config changed
pull-ci-openshift-check-payload-main-verify openshift/check-payload presubmit Ci-operator config changed
Interacting with pj-rehearse

Comment: /pj-rehearse to run up to 5 rehearsals
Comment: /pj-rehearse skip to opt-out of rehearsals
Comment: /pj-rehearse {test-name}, with each test separated by a space, to run one or more specific rehearsals
Comment: /pj-rehearse more to run up to 10 rehearsals
Comment: /pj-rehearse max to run up to 25 rehearsals
Comment: /pj-rehearse auto-ack to run up to 5 rehearsals, and add the rehearsals-ack label on success
Comment: /pj-rehearse list to get an up-to-date list of affected jobs
Comment: /pj-rehearse abort to abort all active rehearsals
Comment: /pj-rehearse network-access-allowed to allow rehearsals of tests that have the restrict_network_access field set to false. This must be executed by an openshift org member who is not the PR author

Once you are satisfied with the results of the rehearsals, comment: /pj-rehearse ack to unblock merge. When the rehearsals-ack label is present on your PR, merge will no longer be blocked by rehearsals.
If you would like the rehearsals-ack label removed, comment: /pj-rehearse reject to re-block merging.

@smith-xyz
Copy link
Copy Markdown
Contributor Author

/pj-rehearse pull-ci-openshift-check-payload-main-payload-scan-4-15 pull-ci-openshift-check-payload-main-payload-scan-4-20 pull-ci-openshift-check-payload-main-payload-scan-5-0

@openshift-merge-bot
Copy link
Copy Markdown
Contributor

@smith-xyz: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

@openshift-ci
Copy link
Copy Markdown
Contributor

openshift-ci Bot commented May 29, 2026

@smith-xyz: all tests passed!

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

@smith-xyz
Copy link
Copy Markdown
Contributor Author

/pj-rehearse ack

@openshift-merge-bot
Copy link
Copy Markdown
Contributor

@smith-xyz: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

@openshift-merge-bot openshift-merge-bot Bot added the rehearsals-ack Signifies that rehearsal jobs have been acknowledged label May 29, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by an approver from all required OWNERS files. rehearsals-ack Signifies that rehearsal jobs have been acknowledged

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant