Skip to content

fix: resolve consumer deadlock on partition rebalance when paused (#681)#736

Open
DmytroLiulchuk wants to merge 1 commit into
Farfetch:masterfrom
DmytroLiulchuk:fix/issue-681-consumer-deadlock
Open

fix: resolve consumer deadlock on partition rebalance when paused (#681)#736
DmytroLiulchuk wants to merge 1 commit into
Farfetch:masterfrom
DmytroLiulchuk:fix/issue-681-consumer-deadlock

Conversation

@DmytroLiulchuk

@DmytroLiulchuk DmytroLiulchuk commented May 18, 2026

Copy link
Copy Markdown

Description
Fixes #681

When a consumer has all partitions paused, KafkaFlow starts a heartbeat background task that calls Consume(1000) in a loop to maintain group membership. If a partition rebalance occurs during this Consume() call, librdkafka invokes the revoke callback (configured via Confluent.Kafka's SetPartitionsRevokedHandler) synchronously on the same thread (documented behavior). The handler calls ConsumerFlowManager.Stop()StopHeartbeat(), which previously awaited the heartbeat task synchronously via _heartbeatTask.GetAwaiter().GetResult(). Since this executes on the heartbeat thread itself, the thread waits for its own completion — a deadlock.

Fix: Remove the synchronous wait in StopHeartbeat(). After Cancel(), the heartbeat task exits naturally on the next loop iteration (while (!IsCancellationRequested) → false), releases the semaphore in finally, and the feeder unblocks.

How Has This Been Tested?

  • Consumer A pauses all partitions (triggers heartbeat)
  • Consumer B joins same group (triggers rebalance during heartbeat's Consume(1000))
  • Without fix: consumer group deadlocks permanently
  • With fix: consumer group recovers and processes new messages

Checklist
 [+] My code follows the style guidelines of this project
 [+] I have performed a self-review of my own code
 [-] I have added tests to cover my changes
 [-] I have made corresponding changes to the documentation

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

[Bug Report]: Consumer deadlocks if there is a partition rebalance while paused

3 participants