Skip to content

fix(tasks): notify workflow service when template stop bulk-stops waiting tasks#3969

Draft
cursor[bot] wants to merge 1 commit into
developfrom
cursor/critical-bug-investigation-af77
Draft

fix(tasks): notify workflow service when template stop bulk-stops waiting tasks#3969
cursor[bot] wants to merge 1 commit into
developfrom
cursor/critical-bug-investigation-af77

Conversation

@cursor

@cursor cursor Bot commented Jun 16, 2026

Copy link
Copy Markdown

Bug and impact

When a user calls Stop all tasks on a template (POST .../templates/{id}/stop_all_tasks) that participates in a workflow, waiting workflow tasks are bulk-stopped in the database without going through finishRun. That means HandleWorkflowTaskCompletion is never called, and the workflow run can remain stuck in running or approval indefinitely.

Trigger scenario:

  1. A workflow run queues a downstream node task (status waiting).
  2. An operator uses Stop all tasks on that template from the template view.
  3. The task is marked stopped in the DB, but the workflow service is never notified.
  4. The workflow run never progresses to a terminal state.

Root cause

StopTasksByTemplate uses a fast path (SetWaitingTasksToStopped) for waiting tasks that bypasses TaskRunner.finishRun, which is the single completion hook that calls HandleWorkflowTaskCompletion. This interaction was introduced with the workflow feature in f29bb3f.

Fix

  • Snapshot waiting workflow tasks before the bulk stop and call HandleWorkflowTaskCompletion for each after they are stopped.
  • Also notify the workflow service for non-async tasks handled in the DB-only cleanup loop (same missing hook).
  • Clear the dispatching flag when the dispatch goroutine exits so a crashed dispatch cannot block the runner reconciler from requeuing an undispatched task.

Validation

  • Added TestTaskPool_StopTasksByTemplate_NotifiesWorkflowOnWaitingTasks
  • Existing TestTaskPool_StopTasksByTemplate_DequeuesWaitingTasksByID still passes
  • Reconciler tests pass
Open in Web View Automation 

…ting tasks

StopTasksByTemplate bulk-updates waiting tasks to stopped without going
through finishRun, so workflow runs never received HandleWorkflowTaskCompletion
and could remain stuck in running/approval until manual intervention.

Also clear the dispatching flag when the dispatch goroutine exits so a
crashed dispatch cannot block the reconciler from requeuing the task.

Co-authored-by: Denis Gukov <fiftin@outlook.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant