A target destroyed between the re-check and the write is classified execute-failed, not a clean abort — so a delete that reached its goal escalates

[genesis-dev-bot]

Found by T5 (issue #194), while defining the outcome of a fault that lands between the precondition re-check and the write. Asserted, not hypothetical: TestFaultBetweenRecheckAndWrite_DestroyingTheTargetIsNotACleanAbort in internal/execute/faultinjection_test.go pins the behaviour that exists today.

The asymmetry

kube.Executor.act (internal/kube/executor.go:449) maps exactly one API server response to a clean-abort sentinel:

if apierrors.IsConflict(err) {
    return nil, fmt.Errorf("%w: ...", ErrPreconditionConflict, ...)
}
return nil, fmt.Errorf("%w: ...", ErrExecute, ...)

execute.Runner then classifies ErrPreconditionConflict as FailureConflict and everything else as FailureExecute, and FailureClass.CleanAbort() is FailureDrifted || FailureConflict. So two faults landing in the same window, on the same object, one millisecond apart, produce opposite verdicts:

Fault in the window	API server	Class	CleanAbort()	What a human sees
pod-failure — the pod moves	409 Conflict	FailureConflict	yes	next cycle re-proposes; nobody is woken
pod-kill — the pod is gone	404 NotFound	FailureExecute	no	escalation

Why the second verdict is wrong for deletepod

FailureExecute's documented meaning is "Whether the change landed may be unknown, which is exactly why the runner stops rather than retrying." Both halves of that are false here:

1. The outcome is not unknown. A 404 on a DELETE carrying a resourceVersion precondition is proof the request was evaluated and nothing was applied — the same certainty a 409 gives.
2. The action's goal has already been reached. The proposal was "delete pod shop/web-dead". The pod is gone. deletePodConverged would return true against the very cluster the runner just gave up on. MaKlaude escalates a remediation that succeeded, by other hands.

This is not a hypothetical race. Under M6 it is a designed one: chaos pod-kill on a pod that a deletepod remediation is in flight against is a scenario the milestone exists to run, and T8 (issue #197) will run it against a live cluster.

Why it wasn't fixed inside T5

Three reasons, in order of weight.

The taxonomy decision is wider than one operation. The same 404 is not obviously benign for the other operations. A vanished Deployment mid-restart, or a vanished Node mid-cordon, is a cluster event a person may well want to hear about even though the write applied nothing. So the fix is not "treat 404 as a clean abort" — it is per-operation, and it needs the operation's goal predicate to decide, which is a structural change to how plan and FailureClass relate.

The right layer is genuinely unclear. Two candidates, and they are not equivalent:

• internal/kube grows a third sentinel (ErrTargetGone) so the transport reports what the API server said and the policy layer decides what it means. Keeps kube free of judgment, which is its stated design property.
• internal/execute inspects apierrors.IsNotFound and consults the plan's convergence predicate against a fresh read before classifying. Keeps the taxonomy where the policy already lives, at the cost of one extra read on a failure path.

Papering over it in T5 would have been the wrong shape. T5's done criteria say the scenario asserts the behaviour that exists and files what is undefined. A silent fix here would have made the timing scenario green while hiding the one finding it produced.

Done criteria

• The write path distinguishes "the target moved" from "the target is gone", rather than folding the second into the generic execute failure.
• For each catalog operation, the vanished-target outcome is a stated decision with a test, not a fallthrough. deletepod reaching its goal state is not an escalation.
• TestFaultBetweenRecheckAndWrite_DestroyingTheTargetIsNotACleanAbort is updated rather than deleted — it carries a guard ("issue #214 has been fixed but this test was not updated") that fires when the classification changes, so the fix cannot land while the scenario still claims the old behaviour.
• docs/remediation.md's account of the abort classes matches.