T8 — End-to-end chaos scenario on kind in CI

[genesis-dev-bot]

Milestone 6, task 8. Plan: issue #188. Depends on T3 (issue #192), T5, T6, T7.

What

The scenarios from T5 and T6 running against a live kind cluster in CI, gating merges.

The negative case is the one that matters most

A cluster not marked eligible, in the same run, receiving no chaos write. A run that only exercises the eligible path proves the feature works and proves nothing about the guarantee.

Where M6 runs (open choice C, answered)

The milestone lands and proves itself on kind. Marking a real cluster eligible is a separate decision I make afterwards, with the mechanism already built and exercised.

So this task is the milestone's proof, and no real cluster is involved.

Note on the local gate

task vet now includes a -tags e2e pass, and TestVetCoversEveryBuildTag (test/devsystem/buildtags_test.go) fails until every custom build tag in the tree has a matching vet pass (PR #189, 1228bc0). If this task introduces a new build tag — plausible, since these scenarios need Chaos Mesh installed and shouldn't run under a bare go test ./... — that guard will require the matching pass, which is the intended behaviour rather than an obstacle.

Done criteria

• Green in CI against kind, and gating merges.
• Each assertion fails the build when violated — verified by breaking it on purpose, not assumed.
• The non-eligible cluster runs in the same job and receives no chaos write.
• Total runtime stays inside the existing e2e budget.

---

Blocked on issue #192 (T3), issue #194 (T5), issue #195 (T6), issue #196 (T7).

[genesis-dev-bot]

Carry-over from T2 (#191), recorded here because T2 is now closed by PR #200.

T2's first done criterion was "an experiment CR is created and deleted on kind under the chaos identity." PR #200 did not meet it literally, and says so rather than quietly narrowing it: Chaos Mesh has no CRDs on the e2e cluster, so .github/workflows/e2e.yml proves the RBAC with SubjectAccessReview POSTs (deliberately not kubectl auth can-i, which would map podchaos through discovery and make an absent resource type the load-bearing detail of every assertion), and the create/delete wire behaviour is covered against a stub apiserver in internal/chaos/injector_test.go. The PR routes the live half here — "whether an experiment actually runs is T8's job".

That's a reasonable split, but T8's done criteria as written don't carry the part that's easy to lose:

• "under the chaos identity" — the scenarios must drive the injector with a rest.Config built for system:serviceaccount:maklaude:maklaude-chaos, not with the CI job's admin kubeconfig. A scenario that creates a real PodChaos as cluster-admin proves Chaos Mesh works and proves nothing about the third identity, and it would pass every assertion in this issue today.
• the delete half — teardown conditions on UID, and NotFound is success with AlreadyAbsent set. Worth asserting against a real object whose controller is actively updating status, since that's exactly the case a resourceVersion precondition would have broken and a stub apiserver doesn't reproduce.

Also pin here what #200 left open: Chaos Mesh can be installed with permission validation, whose webhook checks whether the requester may act on the objects a selector matches. Under that posture maklaude-chaos is refused, because it deliberately holds no pod permissions. Which posture the kind job installs is this task's call; see docs/chaos.md.

[genesis-dev-bot]

Picked this up (in-progress via issues.sh next). Before writing anything: T8 is wider than T7 was, and the width diagnosis on #196 applies to it more than it applied there. Sequencing it the same way, and landing step 1 in this session rather than reading all of it first.

What already exists, so no step re-derives it: the chaos on kind job (T3) builds its own cluster, installs Chaos Mesh through the same scripts/install-chaos-mesh.sh a person runs, asserts the chaos Role against the real CRD both before and after the per-namespace grant, and runs TestE2E_ChaosLifecycle / …ReaperSweepsOnlyItsOwn / …TargetRoleIsTheAllowlist under the maklaude-chaos ServiceAccount kubeconfig, with an always() leak check on the way out. T2's carry-over — "under the chaos identity" and the UID-conditioned delete against a live controller holding a finalizer — is met by buildChaosInjector and awaitAbsent. The installed posture is permission validation on, which is why the target-namespace grant exists and why default is the allowlist's negative control.

What is left, as separate commits in this order:

1. The non-eligible cluster in the same job. This issue says it is the criterion that matters most and it is the only one with no live form at all — internal/chaos/leak_test.go (PR T7 step 1 — a chaos write aimed at a cluster nobody marked must fail the build #222) proves it against stub servers, which proves what MaKlaude composes, not what a real cluster receives. Self-contained: one Go file, no CI change, no new build tag.
2. T5's fault-injection-during-remediation scenario, live. One scenario end to end, not the matrix.
3. T6's correctness scoring over that live scenario — did it fix the fault, should it have been allowed — read from the record rather than from what the job seeded.
4. Break each new assertion on purpose and record what turned red. The done criterion says verified, not assumed, and it is cheap only once the assertions exist.

Runtime budget is a constraint on 2 and 3, not on 1: the chaos job already pays for a cluster and a Chaos Mesh install, and step 1 adds no wait.

Step 1's design, since "same job" is doing real work in that criterion. The physical cluster is the same one either way — eligibility is a marker in MaKlaude's config, not a property the cluster can report — so registering two different clusters would let an environmental difference explain the silence. Instead the same live API server is registered twice, each registration behind its own recording reverse proxy carrying the same chaos credential: marked as maklaude-e2e-chaos, unmarked as maklaude-e2e-unmarked. The unmarked route is proven live first — a probe through it answers from the real API server — so its silence afterwards is caused by the marker and by nothing else, and the marked route takes a real admitted create through its own proxy as the positive control.