Skip to content

fix: a task must not report converged without reaching its declared state - #255

Merged
noahgift merged 1 commit into
mainfrom
fix/task-verifies-completion
Aug 16, 2026
Merged

fix: a task must not report converged without reaching its declared state#255
noahgift merged 1 commit into
mainfrom
fix/task-verifies-completion

Conversation

@noahgift

Copy link
Copy Markdown
Contributor

Closes #254

The defect

completion_check was consulted only as a guard on whether to RUN the command, never re-evaluated afterwards. So converged meant "the command exited 0", not "the resource is in the state it declares" — and the lock recorded that, so the next plan reported no changes over a host that had never converged.

Found in production, not by inspection. paiml/infra's lean-toolchain used sudo: true, which made $HOME=/root, so the Lean toolchain installed where the runner user could not read it:

$ forjar apply -f forjar.yaml -t proof-toolchain --yes
intel: 1 converged, 3 unchanged, 0 failed

$ ssh intel 'command -v lean' ; echo $?
1

Every command in that script succeeded. The purpose was not achieved. Nothing noticed.

The fix

Re-assert the check after the command, and exit non-zero when it still fails — with a message that distinguishes two failures which previously looked identical, except that the second looked like a success:

before after
the command errored failure failure
the command ran and achieved nothing CONVERGED failure, task=not-converged

The check is already written and already cheap; it just ran. Running it once more turns an exit code into a statement about the world.

Tests

tests/falsification_task_verifies_completion.rs, 5 cases, all executing the emitted script under bash rather than pattern-matching it. A script.contains("completion_check") assertion would pass on a script that never runs the check — the class of test that let this through in the first place.

Beyond the regression, they pin what a careless fix would break:

  • a genuinely converging task still passes — or the gate trains people to delete it
  • a failing command is still reported as a command failure, not relabelled as a convergence problem, trading one misleading report for another
  • a task with no completion_check is untouched; this must not become a way to fail tasks that never made the claim
  • the check runs after the command — evaluating it first would pass vacuously for any task whose condition already held, which is exactly the guard semantics being replaced

Mutation-checked: reverting the fix fails a_command_that_succeeds_without_converging_fails and leaves the other four green, so the regression test isolates the defect rather than the suite catching it incidentally.

Note on CI

container_transport fails on this branch with Conflict. The container name "/forjar-integration-test" is already in use. That is not from this change — it is the shared-fixture collision #250 fixes with test_machine_named(), and main does not have that fix yet (git show main:tests/container_transport.rs | grep -c test_machine_named -> 0). Everything else passes: 12,994 tests. It clears once #250 lands.

Behaviour change

Tasks that were silently not converging will now fail. That is the point — they were never converged, and the lock was recording otherwise.

…tate (Refs #254)

`completion_check` was consulted only as a guard on whether to RUN the command,
never re-evaluated afterwards. So `converged` meant "the command exited 0", not
"the resource is in the state it declares" — and the lock recorded that, so the
next `plan` reported `no changes` over a host that had never converged.

Found in production, not by inspection. paiml/infra's `lean-toolchain` used
`sudo: true`, which made $HOME=/root, so the Lean toolchain installed where the
runner user could not read it:

    $ forjar apply -f forjar.yaml -t proof-toolchain --yes
    intel: 1 converged, 3 unchanged, 0 failed
    $ ssh intel 'command -v lean' ; echo $?
    1

Every command in that script succeeded. The purpose was not achieved. Nothing
noticed.

The fix re-asserts the check after the command and exits non-zero when it still
fails, with a message that distinguishes the two failures which previously
looked identical — except that the second looked like a success:

    the command errored                   -> already reported as failure
    the command ran and achieved nothing  -> reported as CONVERGED

The check is already written and already cheap; it just ran. Running it once
more turns an exit code into a statement about the world.

tests/falsification_task_verifies_completion.rs, 5 cases, all EXECUTING the
emitted script under bash rather than pattern-matching it — a
`script.contains("completion_check")` assertion would pass on a script that
never runs the check, which is the class of test that let this through.

Beyond the regression itself, the tests pin the properties a careless fix would
break: a genuinely converging task still passes (or the gate trains people to
delete it); a failing command is still reported as a COMMAND failure, not
relabelled as a convergence problem; a task with no completion_check is
untouched; and the check runs AFTER the command, since evaluating it first
would pass vacuously for any task whose condition already held — which is the
guard semantics being replaced.

Mutation-checked: reverting the fix fails
`a_command_that_succeeds_without_converging_fails` and leaves the other four
green, so the regression test isolates the defect rather than the suite
detecting it incidentally.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@noahgift
noahgift merged commit c74c12c into main Aug 16, 2026
21 checks passed
@noahgift
noahgift deleted the fix/task-verifies-completion branch August 16, 2026 19:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

task reports CONVERGED without re-evaluating its completion_check: a command that exits 0 without reaching the declared state passes

1 participant