Race condition - task never completes on process pool with cancellations

[emaxx-google]

I'm observing that sometimes a task submitted to a ProcessPool never completes, neither with success/failure nor with a timeout. After a lot of debugging the logs show the following sequence of events for the problematic task:

[05:57:55.093] WorkerManager.dispatch: task.id=287736
[05:57:55.100] stop_process: stopped pid=2667133
[05:57:55.138] PoolManager.process_next_message: message=Acknowledgement task.id=287736 worker_id=2667133
// here we're hung, with task id=287736 never completing;
// debug-dumping _pool_manager.task_manager.tasks shows Pebble thinks it's still running on the already-terminated worker:
{287736: Task(id=287736 worker_id=2667133)}

I believe what led to this situation is a race condition between the main process terminating the worker for some previous taskA that was cancel()ed, and the worker actually completing this taskA and moving on to acknowledging a new taskB before actually being killed - leaving the taskB in the limbo state.

[emaxx-google]

Assuming that it's undesired to complicate the worker shutdown process by putting some kind of barriers on it, could we maybe change Pebble to check for and reschedule tasks bound to non-existing workers? (UPD: although if the job is non-idempotent, this would leave a chance of the task breaking by this double-execution.)

[emaxx-google]

Hello, any update or recommendations on this? One of Pebble's core enhancements to Python's standard multiprocessing is the ability of job cancellation, but if this feature causes hung jobs this presents a significant issue.

[noxdafox]

Hello,

apologies for the delay. I am quite swamped.

Could you provide a code snippet which reproduces the issue? I know it cannot be reproduced 100% reliably but I can run it several times until the issue manifests.

I have several stress tests scripts which simulate high loads on highly concurrent environments and randomly timeout or cancel tasks. I run these scripts for several hours before making new releases where critical code path is changed and I have not observed any of such issues over the years.

[noxdafox]

I believe what led to this situation is a race condition between the main process terminating the worker for some previous taskA that was cancel()ed, and the worker actually completing this taskA and moving on to acknowledging a new taskB before actually being killed - leaving the taskB in the limbo state.

The above scenario would work as such:

• Task A is cancelled
• Worker completes task A
• Worker picks up task B
• Worker is terminated
• Pool identifies terminated worker
• Pool compares the worker with the picked up task
• Task B is set to exception: ProcessExpired
• New worker is started

So although the above situation might lead to wrong task being terminated, it would not lead to task B stuck in a limbo state. You can see the logic responsible for this in here:
https://github.com/noxdafox/pebble/blob/master/pebble/pool/process.py#L260

[emaxx-google]

I don't have a standalone reproducer, but the problem was consistently being hit in workloads with like tens of thousands of tasks, so unless the stress-tests are long-running the problem could remain unnoticed. There might be some other timing factor involved.

What I saw in the remote debugger attached to the hung processes were tasks whose worker PIDs were referring to non-existing processes.