Introduce plumtree broadcast method in corrosion. by somtochiama · Pull Request #492 · superfly/corrosion

somtochiama · 2026-06-24T14:44:45Z

This pull requests adds a new plumtree protocol for broadcasting changes across a cluster. It is modelled after the original plumtree protocol outlined in the original paper but it differs in a number of key ways.

Full membership: Corrosion's plumtree protocol is built top of SWIM, a full membership protocol while the paper uses hyparview which is a partial membership. A node knows about every node in the cluster, and randomly selects eager and lazy peers from known peers.
Multi-sender: The original plumtree protocol is ideal for single-sender trees but we have multi-master setup where every node can disseminate changes. To make the tree stable under many concurrent senders:
- We tolerate a configurable number of duplicates: a PRUNE is only sent once a duplicate count crosses prune_threshold, rather than on the first duplicate. We trade a little duplication for tree stability, since we're trying to find peers that are good sources across multiple senders. This also has the additional benefit that each node has multiple links for a particular sender so that a failed node doesn't
- Links are asymmetric: receiving a gossip from a peer does not add them to our eager set. A peer is only promoted to eager when it deliberately sends a GRAFT.
Fanout: We use a much wider fanout than recommended in the paper, while the paper recommends a fanout of five we adjust based on the current number of nodes in the cluster to (log N * 3). This is mostly because we are aiming for lower latency, and can tolerate some duplication.
RTT-aware peer selection: Rather than picking eager and lazy peers at random, we bias toward low-RTT peers for the eager set, with a handful of far ones mixed in so messages still spread quickly across clusters that aren't close to each other. This would still get adjusted via prunes and grafts by the protocol.

The plumtree method can be enabled by adding this to the node config

[gossip]
broadcast_method = "plumtree"

This was partly inspired by: https://github.com/johnnywale/memberlist-plumtree

Signed-off-by: Somtochi Onyekwere <somtochionyekwere@gmail.com>

gorbak25

Amazing work! Mostly nits. The main thing i'm worrying about is that there are no tests for rebalancing the tree when a peer gets down/shuts down ungracefully. I don't see other peers getting notified that we're down on graceful shutdown.

gorbak25 · 2026-07-05T17:39:34Z

+
+Lower values optimize the tree more aggressively (more `GRAFT`/`PRUNE` churn); higher values leave the tree alone unless the improvement is large. Defaults to `7`.
+
+##### `batch_gossip`


Does it seem that we will use that option?

Yes, Batching gossips didn't help smaller regional clusters. But I think it would be useful on global clusters that is pushing a lot of changes.

gorbak25 · 2026-07-06T07:36:16Z

+        let (near_pool, mid_pool, far_pool) = self.bucket_pools();
+
+        // we mostly want eager peers to be low rtt, but throw in a mix of mid and far peers;
+        // percentage is roughly 50% near, 30% mid, 20% far. targets are computed over the


Can we make that ratio configurable and exposed in the config?

That sounds good

gorbak25 · 2026-07-06T07:40:32Z

@@ -0,0 +1,2209 @@
+use indexmap::{IndexMap, IndexSet};


Amazing work! The main thing which is missing is to send PRUNE's to eager peers during graceful node shutdown. Imagine we run systemctl stop corrosion2 on an node which has a lot of links. Without graceful handling we would need to first wait for foca to decect the peer is down, then plumtree will remove that peer and only after a few missed rounds the tree will get rebalanced. What is the round interval?

fair, Mostly left membership notifications to foca.

Because tree is not symmetric - we could have eager peers that don't have us an eager peer. So prune only saves them from sending us a message. We could maintain a separate set with this info though, although foca is quick enough that I don't think that it is worth it.

plumtree will remove that peer and only after a few missed rounds the tree will get rebalanced.

Any missed messages would cause grafts before foca even sends the notifications.

gorbak25 · 2026-07-06T07:44:18Z

+}
+
+#[derive(Debug, Clone, Serialize, Deserialize)]
+pub struct PlumtreeConfig {


round interval?

Do you mean number of rounds when we keep broadcasting a message?

gorbak25 · 2026-07-06T07:45:54Z

+#[derive(Debug, Clone, Default, Serialize, Deserialize)]
+pub struct PlumtreeStats {
+    /// Peers we push GOSSIP to directly (the spanning-tree branches).
+    pub eager_peers: Vec<ActorId>,


OFFTOP: I'm tempted to vibe code a visualization of this :D Imagine we would query those edges every few seconds and we could lookup how the tree looks like

Ha, that's a very good idea. We could easily see the tree and how it is changing.

gorbak25 · 2026-07-06T07:55:09Z

+            counter!("corro.plumtree.send.dropped").increment(1);
+        }
+    }
+


prune eager links to other peers or notify them that we're down?

Signed-off-by: Somtochi Onyekwere <somtochionyekwere@gmail.com>

somtochiama requested a review from gorbak25 June 24, 2026 14:45

somtochiama added 4 commits July 3, 2026 20:41

plum-foca: plumtree broadcast library

f43d035

corro-agent: wire plumtree into broadcast path

3144483

plum-foca: broadcast simulation tests

256473c

docs for plumtree settings

232917e

Signed-off-by: Somtochi Onyekwere <somtochionyekwere@gmail.com>

somtochiama force-pushed the somto/test-bcast branch from 796e90a to 232917e Compare July 3, 2026 19:50

bump confirm

6b4a200

Signed-off-by: Somtochi Onyekwere <somtochionyekwere@gmail.com>

gorbak25 requested changes Jul 6, 2026

View reviewed changes

somtochiama added 8 commits July 6, 2026 14:22

update plumtree docs, and sim tests

abfe3df

Signed-off-by: Somtochi Onyekwere <somtochionyekwere@gmail.com>

use duplicates per message instead of rmr

4f011c5

Signed-off-by: Somtochi Onyekwere <somtochionyekwere@gmail.com>

add docs for plumtree messages

684202a

Signed-off-by: Somtochi Onyekwere <somtochionyekwere@gmail.com>

docs for plumtreeState

6ac88d0

Signed-off-by: Somtochi Onyekwere <somtochionyekwere@gmail.com>

send prune on shutdown

637cf82

Signed-off-by: Somtochi Onyekwere <somtochionyekwere@gmail.com>

use weak sender in metrics loop

04e8ff8

Signed-off-by: Somtochi Onyekwere <somtochionyekwere@gmail.com>

make eager ratios configurable

66e1f2b

Signed-off-by: Somtochi Onyekwere <somtochionyekwere@gmail.com>

add failures in plumtree sim tests

4b66697

Signed-off-by: Somtochi Onyekwere <somtochionyekwere@gmail.com>


		Lower values optimize the tree more aggressively (more `GRAFT`/`PRUNE` churn); higher values leave the tree alone unless the improvement is large. Defaults to `7`.

		##### `batch_gossip`

Uh oh!

Conversation

somtochiama commented Jun 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gorbak25 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

somtochiama Jul 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

somtochiama commented Jun 24, 2026 •

edited

Loading

somtochiama Jul 6, 2026 •

edited

Loading