Skip to content

fix: exclude dead workers from reassignment to break orchestrator deadlock#344

Open
Matthew-Selvam wants to merge 1 commit into
Conway-Research:mainfrom
Matthew-Selvam:fix/266-259-orchestrator-deadlock-v2
Open

fix: exclude dead workers from reassignment to break orchestrator deadlock#344
Matthew-Selvam wants to merge 1 commit into
Conway-Research:mainfrom
Matthew-Selvam:fix/266-259-orchestrator-deadlock-v2

Conversation

@Matthew-Selvam

Copy link
Copy Markdown

Summary

Fixes #266, fixes #259. The orchestrator's stale-task recovery (handleExecutingPhase) resets a dead worker's task back to pending when isWorkerAlive() says the worker is gone — but never updates the worker's own status. Meanwhile findBusyAgentForReassign() only excludes idle agents when picking a busy worker to hand a freed task to, not dead ones. So the very next matching pass hands the recovered task straight back to the same dead worker. It dies again, gets "recovered" again, and reassigned again — an infinite recover-reassign-die loop that never completes the goal. That's what leaves orchestrator.state permanently stuck in executing and blocks all new goal creation (both #266 and #259 describe this from different angles — a low-RAM VM where workers OOM, and a stale local:// worker after a restart).

Fix

  • When stale-task recovery detects a dead worker, mark its children row 'dead' and call agentTracker.updateStatus(address, "dead"), so it's excluded from future reassignment.
  • Defense-in-depth: findBusyAgentForReassign() now also checks the existing isWorkerAlive() hook directly, in case a task ever ends up assigned to a dead worker through some other path than the recovery block above.

Testing

  • tsc --noEmit passes.
  • Added two regression tests:
    • findBusyAgentForReassign (via matchTaskToAgent) falls through to self-assignment rather than handing a task back to a worker isWorkerAlive reports dead.
    • The stale-task recovery path in handleExecutingPhase (via tick()) marks the dead worker's children row and tracker status, and — since the worker is now excluded — the same tick reassigns the freed task off it instead of looping back to it.
  • Full orchestration test suite: 286/286 passing (orchestrator.test.ts, messaging.test.ts, task-graph.test.ts, orchestrator-harness.test.ts, local-worker-harness.test.ts, attention.test.ts, workspace.test.ts, local-worker-security.test.ts).

…dlock

The stale-task recovery in handleExecutingPhase() reset a dead worker's
task back to 'pending' but never touched the worker's own status. Since
findBusyAgentForReassign() only excludes *idle* agents (not dead ones)
when picking a busy worker to reassign a freed task to, the very next
matching pass would hand the task straight back to the same dead worker.
That worker dies again next tick, gets 'recovered' again, and reassigned
again — an infinite recover-reassign-die loop that never completes the
goal, which is what left orchestrator state permanently stuck in
'executing' and blocked all new goal creation.

Fix: when a dead worker is detected during stale-task recovery, mark its
children row 'dead' and update the agent tracker to match, so it's
excluded from future reassignment. Also adds a defense-in-depth check in
findBusyAgentForReassign() itself, via the existing isWorkerAlive() hook,
in case a task ever ends up assigned to a dead worker through some other
path.

Adds regression tests: one confirming findBusyAgentForReassign() falls
through to self-assignment rather than handing a task back to a
worker isWorkerAlive() reports dead, and one confirming the stale-task
recovery path in handleExecutingPhase() marks the dead worker's children
row and tracker status so the same tick reassigns off it rather than
looping back to it.

Fixes Conway-Research#266, Fixes Conway-Research#259
@Matthew-Selvam

Copy link
Copy Markdown
Author

This is ready for maintainer review whenever you get a chance — happy to make any changes requested.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

1 participant