fix(tasks): notify workflow service when template stop bulk-stops waiting tasks#3969
Draft
cursor[bot] wants to merge 1 commit into
Draft
fix(tasks): notify workflow service when template stop bulk-stops waiting tasks#3969cursor[bot] wants to merge 1 commit into
cursor[bot] wants to merge 1 commit into
Conversation
…ting tasks StopTasksByTemplate bulk-updates waiting tasks to stopped without going through finishRun, so workflow runs never received HandleWorkflowTaskCompletion and could remain stuck in running/approval until manual intervention. Also clear the dispatching flag when the dispatch goroutine exits so a crashed dispatch cannot block the reconciler from requeuing the task. Co-authored-by: Denis Gukov <fiftin@outlook.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Bug and impact
When a user calls Stop all tasks on a template (
POST .../templates/{id}/stop_all_tasks) that participates in a workflow, waiting workflow tasks are bulk-stopped in the database without going throughfinishRun. That meansHandleWorkflowTaskCompletionis never called, and the workflow run can remain stuck inrunningorapprovalindefinitely.Trigger scenario:
waiting).stoppedin the DB, but the workflow service is never notified.Root cause
StopTasksByTemplateuses a fast path (SetWaitingTasksToStopped) for waiting tasks that bypassesTaskRunner.finishRun, which is the single completion hook that callsHandleWorkflowTaskCompletion. This interaction was introduced with the workflow feature in f29bb3f.Fix
HandleWorkflowTaskCompletionfor each after they are stopped.dispatchingflag when the dispatch goroutine exits so a crashed dispatch cannot block the runner reconciler from requeuing an undispatched task.Validation
TestTaskPool_StopTasksByTemplate_NotifiesWorkflowOnWaitingTasksTestTaskPool_StopTasksByTemplate_DequeuesWaitingTasksByIDstill passes