Skip to content

PKG-1433: Bake PPG package-test snapshots on Hetzner (Rocky, Alma, OL) - #4323

Open
nogueiraanderson wants to merge 4 commits into
masterfrom
PKG-1433-hetzner-image-factory
Open

PKG-1433: Bake PPG package-test snapshots on Hetzner (Rocky, Alma, OL)#4323
nogueiraanderson wants to merge 4 commits into
masterfrom
PKG-1433-hetzner-image-factory

Conversation

@nogueiraanderson

@nogueiraanderson nogueiraanderson commented Aug 7, 2026

Copy link
Copy Markdown
Contributor

Feature

  • Hetzner snapshot factory for PPG package tests: bake, fresh-boot smoke with a real PPG install, label-flip promote, prune-based retention, weekly rebake
  • Molecule consumer path (delegated-driver env exporter, token binding, pilot job) and an hourly leak janitor

Why

  • PG QA is evaluating Hetzner as the primary substrate for package-test VMs with AWS as fallback
  • Rocky/Alma seed from official Hetzner images; OL has none, so bases are rescue-uploaded once and the lineage chains from them
  • Seven x86_64 combos are baked, smoke-passed and promoted (Rocky/Alma 8/9/10 + OL9). The OL8/OL10 rows follow once their bases are bootstrapped. The pilot ran the rocky-9 scenario green three times with zero leaks

Tickets

@nogueiraanderson
nogueiraanderson requested a review from a team as a code owner August 7, 2026 07:28
Comment thread ppg/packer-hetzner/justfile Outdated
Comment thread ppg/packer-hetzner/justfile Outdated
Comment thread ppg/packer-hetzner/smoke/smoke.pkr.hcl
Comment thread .github/workflows/ppg-hcloud-janitor.yml Outdated
Comment thread ppg/packer-hetzner/scripts/janitor.sh Outdated
Comment thread .github/workflows/ppg-hcloud-factory.yml Outdated
Comment thread .github/workflows/ppg-hcloud-factory.yml Outdated
Comment thread ppg/packer-hetzner/smoke/smoke.pkr.hcl Outdated
Comment thread vars/moleculeEnvPPGHetzner.groovy Outdated
@nogueiraanderson
nogueiraanderson force-pushed the PKG-1433-hetzner-image-factory branch 2 times, most recently from 331d12b to abb4792 Compare August 7, 2026 13:45
- ppg/packer-hetzner: packer-plugin-hcloud refresh + seed templates,
  12-combo design (7 active x86 rows until OL8/10 seeds and CAX stock),
  label-selector lineage chaining requiring source=factory + smoke=passed
- fresh-boot smoke via shared smoke-run.sh: sentinel pair gates candidate
  deletion to genuine provisioner failures, smoke=passed labeled pre-promote
- promote.sh with fail-closed role/env guard, prune.sh keep-4 retention
  with a two-tier stale sweep, janitor with namespaced selectors and
  honest failure semantics
- GHA workflows: tokenless PR check, bake matrix, weekly cron + self-prune,
  6h-apply janitor cron with Slack alerts, environment-scoped token jobs,
  setup-hcloud composite action, prod-serialized concurrency
- OL base bootstrap (rescue-mode qcow2 upload, auto raw+xz conversion,
  EL10-safe cloud-init check)
- molecule consumer vars (delegated driver, fail-closed selector) and the
  ppg-hetzner-poc pilot job (param validation, scoped credentials,
  per-workspace pinned collections with the hcloud datacenter guard)
@nogueiraanderson
nogueiraanderson force-pushed the PKG-1433-hetzner-image-factory branch from abb4792 to 3ce1216 Compare August 7, 2026 13:46
- smoke-run.sh: fail-closed identity guard (promote.sh pattern) before the
  boot and again before the candidate delete, so a wrong or relabeled
  snapshot ID is never booted or deleted
- refresh/smoke templates: ssh_keys_labels on the packer temp keys; janitor
  sweeps the two new roles so keys from killed runs cannot leak forever
- tests/: stubbed hcloud+packer suites for both behaviors, run by check.sh
- Bake, prune, and janitor jobs assume a master-pinned GitHub-OIDC AWS role
  and read the Hetzner token from the /ppg/hcloud-factory-token SSM
  parameter, so no Hetzner credential lives in GitHub and a branch-ref
  dispatch fails at AssumeRole
- Role + parameter are Terraform in percona-cd-platform
  (iam-gha-ppg-hcloud-factory.tf)
- README token section updated; comment trims
- Tool-setup actions run before authentication; AWS credentials and the
  Hetzner token never enter the job env (step outputs, wired explicitly
  into each consuming step)
- Janitor Slack alert moves to a credential-free job fed by sweep outputs
- Fetch step rejects the SSM placeholder, role sessions capped at 900s,
  checkouts set persist-credentials: false
- Bake gate is an explicit event allowlist, mutable role-ARN override
  dropped
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants