Fix Windows orchestrator test race condition#311
Open
circleci-app[bot] wants to merge 1 commit into
Open
Conversation
Poll for expected task unclaims/events before asserting, to prevent the test from racing with the orchestrator's final updates on Windows. On Windows, when the task agent process exits with a fatal error, there is a race condition between the orchestrator sending the failure event to the runner API and the test assertions checking for those events. The orchestrator's error handling happens asynchronously, and on Windows the process cleanup is synchronous via Job Objects, which can cause tight timing windows. This fix adds polling with a 30-second timeout to wait for the expected number of task events and unclaims before performing the assertions. This ensures the test doesn't fail due to timing issues while still catching genuine failures. Fixes the "error: task agent encountered fatal error" test failure on Windows where the test expected a TaskEvent but received nil due to the race condition.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR fixes a race condition in the orchestrator tests that was causing failures on Windows, specifically in the test case "error: task agent encountered fatal error".
Problem
The test was failing on Windows with the following error:
The orchestrator was unable to send the failure event to the fake runner API, with connection refused errors occurring across 16 retry attempts over 48 seconds.
Root Cause
On Windows, there's a race condition between:
Windows uses Job Objects for process cleanup, which is synchronous and creates tight timing windows. The test was asserting before the orchestrator had completed sending the failure event.
Solution
This PR adds polling logic to wait for the expected number of task events and unclaims (with a 30-second timeout) before performing the assertions. This approach:
Testing
Original failing job
Agent run