Local shovel ignores queue-level backpressure from quorum queues and streams

[lukebakken]

Problem

The local shovel (rabbit_local_shovel) does not handle {block, QName} actions returned by rabbit_queue_type:deliver/4. This means it ignores queue-level backpressure from quorum queues and stream queues, continuing to publish into an overloaded queue until the condition escalates to a system-wide disk or memory alarm.

Line 698 of rabbit_local_shovel.erl (on main) has an explicit TODO:

     %% TODO handle {block, QName}
     (_Action, S0) ->
          S0

Background

When a quorum queue's rabbit_fifo_client exceeds its soft limit (32 pending Raft commands), it returns {block, QName} as an action from enqueue. Later, when applied commands reduce the pending count below the soft limit, it returns {unblock, QName} via handle_ra_event. Stream queues use the same mechanism (soft limit 256).

In rabbit_channel, these actions invoke credit_flow:block/1 and credit_flow:unblock/1, which throttle the reader process and ultimately block the publishing connection.

The local shovel does not use credit flow. It has its own blocking mechanism via rabbit_alarm registration, an alarms set, a pending_delivery buffer, and an is_blocked/1 check before each forward.

Root Cause

handle_dest_queue_actions/2 handles {settled, ...} and {rejected, ...} but drops {block, QName} and {unblock, QName} actions silently.

Classic queues are unaffected because they never return {block, ...} actions - they use only credit flow for backpressure, and always return an empty actions list from deliver/3.

Impact

When the local shovel publishes to a quorum queue or stream that is under heavy load (Raft pipeline backed up, pending commands above soft limit), the shovel continues forwarding at full rate. This can:

• Grow the Raft log unboundedly on the leader and followers
• Increase memory pressure from accumulated pending commands
• Eventually trigger a system-wide memory alarm (at which point the shovel's alarm handler finally kicks in)

The gap is the time between "queue is struggling" and "system-wide alarm fires." During that window, the shovel provides no backpressure.

Affected Module

• deps/rabbitmq_shovel/src/rabbit_local_shovel.erl

Implementation Constraints

A naive buffer-and-drain approach (track {block, QName} in a set, buffer messages, drain on {unblock, QName}) does not work due to several interacting constraints:

1. {unblock, ...} only arrives in on_confirm mode

handle_dest({queue_event, ...}) only matches #{ack_mode := on_confirm}. In on_publish and no_ack modes, {queue_event} messages for the destination queue are never processed by the dest handler. This means {unblock, ...} is never received, and the shovel gets permanently stuck.

2. Async drain deadlock

forward_pending_delivery runs synchronously, pipelining Ra commands in a tight loop. After 32 messages, {block, ...} fires again. The corresponding {unblock, ...} only arrives after Ra asynchronously applies those commands and sends back a {queue_event}. If the drain has already returned, no code path re-triggers it until the next {queue_event} arrives - and if all pending Ra commands are applied in a single batch, only one {unblock, ...} is generated, causing the drain/block cycle to stall.

3. forward_pending_delivery bypasses remaining_unacked

forward_pending_delivery calls do_forward directly, not forward/3. This skips the remaining_unacked := 0 guard that implements src-delete-after (autodelete). Draining buffered messages can overshoot the autodelete boundary.

Suggested Fix

The correct approach is likely to pause the source consumer rather than buffer at the dest side:

• On {block, QName}: withdraw link credit from the source consumer (or cancel it)
• On {unblock, QName}: re-grant credit (or re-subscribe)

This avoids buffering entirely - backpressure is applied at the source, which is analogous to how rabbit_channel handles it (via credit flow to the reader). This approach interacts with the source credit mechanism (maybe_grant_credit) and the consumer lifecycle.

For on_publish and no_ack modes, where {queue_event} messages are not processed for the destination, a different mechanism is needed - perhaps registering a separate event handler for the dest queue state, or forwarding with the flow option so that credit flow applies.