Force QQ deletions can cause RA member leaks

[Ayanda-D]

Describe the bug

We are seeing Ra Member Leaks, when normal QQ deletion fails (due to e.g. network glitches) and falls back to rabbit_quorum_queue:force_delete_queue/1. This leads to indefinite queue declare failures for the same queue name (whose Ra Member remnants from the previous incarnation still exist). An example of a sequence of events:

1. Failed Normal Quorum Queue Delete

2026-08-03 02:22:25.467893+00:00 [warning] <0.27762.0> Could not delete quorum 'queue 'xxxxxx' in vhost 'xxxxxx'', not enough nodes  online to reach a quorum: [{error,noproc},
2026-08-03 02:22:25.467893+00:00 [warning] <0.27762.0> {error,noproc},
2026-08-03 02:22:25.467893+00:00 [warning] <0.27762.0> {timeout,
2026-08-03 02:22:25.467893+00:00 [warning] <0.27762.0> {'xxxxxx',
2026-08-03 02:22:25.467893+00:00 [warning] <0.27762.0> 'rabbit@host-111'}},
2026-08-03 02:22:25.467893+00:00 [warning] <0.27762.0> {timeout,
2026-08-03 02:22:25.467893+00:00 [warning] <0.27762.0> {'xxxxxx',
2026-08-03 02:22:25.467893+00:00 [warning] <0.27762.0> 'rabbit@host-222'}},
2026-08-03 02:22:25.467893+00:00 [warning] <0.27762.0> {timeout,
2026-08-03 02:22:25.467893+00:00 [warning] <0.27762.0> {'xxxxxx',
2026-08-03 02:22:25.467893+00:00 [warning] <0.27762.0> 'rabbit@host-333'}},
2026-08-03 02:22:25.467893+00:00 [warning] <0.27762.0> {timeout,
2026-08-03 02:22:25.467893+00:00 [warning] <0.27762.0> {'xxxxxx',
2026-08-03 02:22:25.467893+00:00 [warning] <0.27762.0> 'rabbit@host-444'}},
2026-08-03 02:22:25.467893+00:00 [warning] <0.27762.0> {error,noproc}]. Attempting force delete.

2. The force delete loops through all servers, and attempts delete. Force delete is best-effort operation, (no quorum) and removes the metadata record but leaks Ra member processes on remote nodes

https://github.com/rabbitmq/rabbitmq-server/blob/v4.3.3/deps/rabbit/src/rabbit_quorum_queue.erl#L1165-L1176

Example, in this case the force deletion takes ~77seconds to complete across the cluster (with broker under load)

  2026-08-03 02:22:31.119931+00:00 [info] <0.245132098.0> Deleting server 'xxxxxx' and its data directory. (on local node)
  2026-08-03 02:22:34.944956+00:00 [info] <15423.205710444.0> Deleting server 'xxxxxx' and its data directory. (on remote node)
  2026-08-03 02:23:00.664541+00:00 [info] <15426.214931516.0> Deleting server 'xxxxxx' and its data directory. (on remote node)
  2026-08-03 02:22:59.208212+00:00 [info] <15425.53715023.0> Deleting server 'xxxxxx' and its data directory. (on remote node)
  2026-08-03 02:23:05.745231+00:00 [info] <0.245163399.0> Deleting server 'xxxxxx' and its data directory. (on local node)
  2026-08-03 02:23:05.745598+00:00 [info] <15427.231906357.0> Deleting server 'xxxxxx' and its data directory. (on remote node)
  2026-08-03 02:23:05.759928+00:00 [info] <15428.29999002.2> Deleting server 'xxxxxx' and its data directory. (on remote node)
  2026-08-03 02:23:05.845455+00:00 [info] <0.245165687.0> Deleting server 'xxxxxx' and its data directory. (on local node)
  2026-08-03 02:23:27.552543+00:00 [info] <0.245204696.0> Deleting server 'xxxxxx' and its data directory. (on local node)
  2026-08-03 02:23:42.704280+00:00 [info] <0.245183670.0> Deleting server 'xxxxxx' and its data directory. (on local node)

In this case, cleanup starts at 02:22:31 finally settles at 02:23:42. The force cleanup (which has no consensus) had to reach each member node individually, which took over 77 seconds to complete across 5+ nodes. Under inbound load, this caused the race window.

3. All subsequent queue declarations fail indefinitely with reason already_started as below.

2026-08-03 10:57:25.928950+00:00 [error] <0.17656147.1> ra: failed to start a server #{id => {'xxxxx','rabbit@host-333'},machine => .....},tick_timeout => 5000}, error: {error,
2026-08-03 10:57:25.928950+00:00 [error] <0.17656147.1> {shutdown,
2026-08-03 10:57:25.928950+00:00 [error] <0.17656147.1> {failed_to_start_child,
2026-08-03 10:57:25.928950+00:00 [error] <0.17656147.1> 'xxxxx',
2026-08-03 10:57:25.928950+00:00 [error] <0.17656147.1> {already_started,
2026-08-03 10:57:25.928950+00:00 [error] <0.17656147.1> <15428.205774787.0>}}}}

and eventually giving up with: "No servers were successfully started" → {error, cluster_not_formed} - this never succeeds (until manual intervention cleanup the leaked Ra components)

4. ra:start_cluster/1 treats this as fatal and calling declare_queue_error which issues a protocol_error back to the client (for every failed re-declaration attempt at this point and beyond).

5. As more evidence of Ra member leaks, expiry timer are seen executing for non-existing queues (from rabbit_fifo:tick/2 effects), and failing the assertion:

2026-08-03 10:57:29.136308+00:00 [error] <0.9113570.1> Error in process <0.9113570.1> on node 'xxxxx@host-555' with exit value:
2026-08-03 10:57:29.136308+00:00 [error] <0.9113570.1> {{badmatch,{error,not_found}},
2026-08-03 10:57:29.136308+00:00 [error] <0.9113570.1>  [{rabbit_quorum_queue,'-spawn_deleter/1-fun-0-',1,
2026-08-03 10:57:29.136308+00:00 [error] <0.9113570.1>                        [{file,"rabbit_quorum_queue.erl"},{line,578}]}]}
2026-08-03 10:57:29.136308+00:00 [error] <0.9113570.1>

Reproduction steps

The same repeat queue.declare failures/errors can be easily reproduced with

1. Declare a Quorum Queue with x-expires argument
2. Delete its entry in metadata before it expires ( ctl eval 'rabbit_amqqueue:internal_delete(Q, <<"internal-user">>).' )
3. Verify RA queue entry exists ( `ctl eval 'ra_leaderboard:overview().' )
4. Attempt to re-declare the same queue
5. Check logs for error:

2026-08-06 17:34:46.535330+01:00 [error] <0.1082.0> ra: failed to start a server #{id => .....}, error:
{error, {shutdown, {failed_to_start_child, '%2F_TEST', {already_started, <0.989.0>}}}}

6. Check logs for repeat exipry crashes for non-existent QQ in metadata

2026-08-06 17:36:09.333569+01:00 [error] <0.1125.0> Error in process <0.1125.0> on node rabbit@K3K70TDT7R with exit value:
2026-08-06 17:36:09.333569+01:00 [error] <0.1125.0> {{badmatch,{error,not_found}},
2026-08-06 17:36:09.333569+01:00 [error] <0.1125.0>  [{rabbit_quorum_queue,'-spawn_deleter/1-fun-0-',1,
2026-08-06 17:36:09.333569+01:00 [error] <0.1125.0>                        [{file,"rabbit_quorum_queue.erl"},{line,659}]}]}

Once these conditions exist in the broker/cluster, queue declaration fails indefinitely causing outages until the RA components are manually cleaned up.

Expected behavior

We expect thorough cleanup of QQ metadata and all RA components, even after failed normal/force deletes.

This could be via best-effort retries, OR in an async event issued when rabbit_quorum_queue:force_delete_queue/1 is triggered, and an event handler/listener process which follows up and retries to ensure the queue is eventually cleaned-up across the cluster.

Additional context

No response