Skip to content

doc: add centralized chassis IPC over ZeroMQ HLD#2438

Open
pkudlurr1 wants to merge 1 commit into
sonic-net:masterfrom
pkudlurr1:centralized-chassis-ipc-hld
Open

doc: add centralized chassis IPC over ZeroMQ HLD#2438
pkudlurr1 wants to merge 1 commit into
sonic-net:masterfrom
pkudlurr1:centralized-chassis-ipc-hld

Conversation

@pkudlurr1

@pkudlurr1 pkudlurr1 commented Jul 2, 2026

Copy link
Copy Markdown

Summary

This PR adds a High-Level Design (HLD) document for the Zmq + Redis IPC fabric on a SONiC centralized chassis — where central Redis databases on the Supervisor must deliver table updates to per-NPU orchagent instances on every line card without scaling limits of direct Redis pub/sub fan-out.

The document is intended for community review as part of the centralized chassis design effort, complementing the routing HLD and aggregate ACK HLD.

What this HLD covers

Problem and solution

  • PST/CST scaling limit — standard ProducerStateTable / ConsumerStateTable supports at most one consumer per table channel; multi-NPU line cards cannot share the same Redis notification path
  • Remote Redis pub/sub cost — direct Supervisor Redis subscriptions do not scale with line-card and NPU count
  • Hybrid Redis + ZeroMQ path — producers commit to Redis (central store) and publish the same update on the midplane in one call
  • Hub-and-Spoke proxy — Supervisor-resident XPUB/XSUB message proxy decouples publishers from subscribers with constant per-frame fan-out cost

Architecture

  • zmqEnhanced producer / consumer — extensions to PST/CST for IPC-carried tables on Supervisor and line cards
  • Midplane message format — frame envelope and batched-entry layout
  • Full database sync — cold-joining consumers reconcile from central Redis before draining live ZMQ deltas

Operational flows

  • Steady-state update, batched update with per-key coalescing, full database sync, and failure/recovery scenarios
  • Back-pressure and saturation queue behavior (Figure 6)

Performance characterization

  • Redis-only baseline vs hybrid Zmq + Redis convergence comparison at route scale

Files added

Path Description
doc/modular-chassis-voq/centralized_ipc_hld.md Main HLD document
doc/modular-chassis-voq/img/centralized_ipc_*.png Architecture, operational flow, and queueing figures
doc/modular-chassis-voq/img/zmq_redis_convergence_timing.png Convergence timing comparison figure
doc/modular-chassis-voq/img/*.py, install_figures.sh Figure generation scripts

Reference documents

Document Link
Centralized chassis baseline voq_chassis_hld.md
Centralized chassis routing HLD PR #2382
Aggregate ACK mechanism HLD PR #2408
VOQ architecture architecture.md
VOQ HLD voq_hld.md

Test plan

  • Review HLD scope, terminology (Supervisor / line card), and IPC-carried table set against routing HLD and VOQ docs
  • Verify diagram links and image assets render correctly on GitHub
  • Confirm cross-references to related HLDs are accurate
  • Community review in routing / centralized chassis working group

@linux-foundation-easycla

linux-foundation-easycla Bot commented Jul 2, 2026

Copy link
Copy Markdown

CLA Not Signed

@mssonicbld

Copy link
Copy Markdown
Collaborator

/azp run

@azure-pipelines

Copy link
Copy Markdown
No pipelines are associated with this pull request.

@pkudlurr1 pkudlurr1 force-pushed the centralized-chassis-ipc-hld branch from 13c8131 to 2163efc Compare July 2, 2026 00:18
@mssonicbld

Copy link
Copy Markdown
Collaborator

/azp run

@azure-pipelines

Copy link
Copy Markdown
No pipelines are associated with this pull request.

@mssonicbld

Copy link
Copy Markdown
Collaborator

/azp run

@azure-pipelines

Copy link
Copy Markdown
No pipelines are associated with this pull request.

@pkudlurr1 pkudlurr1 force-pushed the centralized-chassis-ipc-hld branch from f3c2049 to fd3c240 Compare July 2, 2026 01:07
@mssonicbld

Copy link
Copy Markdown
Collaborator

/azp run

@azure-pipelines

Copy link
Copy Markdown
No pipelines are associated with this pull request.

Add the modular-chassis VoQ IPC HLD with architecture and operational
flow figures, §7.1 frame envelope diagram, expanded table of contents,
and figure generation scripts.
@pkudlurr1 pkudlurr1 force-pushed the centralized-chassis-ipc-hld branch from fd3c240 to 587009d Compare July 2, 2026 01:20
@mssonicbld

Copy link
Copy Markdown
Collaborator

/azp run

@azure-pipelines

Copy link
Copy Markdown
No pipelines are associated with this pull request.

@eddieruan-alibaba

Copy link
Copy Markdown
Contributor
  1. Can you give a write up to compare two approaches a. make this IPC between fpmsyncd and orchagent, b. make this IPC between orchagent and syncd. The reason is that currently you are using 1 stage forwarding, ingress LC will make full encap, egress LC would just a pass through. The second approach would make you have less chance in retry right?

  2. Cisco forwarding moved from 1 stage, to 2 stage, then to 1.5 stage forwarding due to various scale considerations. Can you make a table to compare pros / cons among distributed voq chassis, Ethernet based chassis. and centralized voq chassis. Basically, let's articulate clearly on the benefits we would get, since this centralized approach would have large code changes, we need to make sure we understand the benefits for this investments.

  3. "any message loss on the midplane path" Any message loss in midplane is fatal. How do we flag it out for none OIR cases? "Midplane notification is best-effort, not guaranteed" if this is best-effort, is there any insurance that this would happen in a very small probability for none OIR cases. For example, would midplane have a dedicated physical path for communication?

  4. At infra level, how do we insure the delivery order between ingress / egress LCs. For example in ECMP, LAG cases, when adding a path, how do we make sure egress is ready before programming ingress and vise versa for removing a path.

  5. I am confused on your two claim "Reachable IPC transports — Producers on the Supervisor and zmqEnhanced consumers on line cards SHALL reach the same XPUB/XSUB proxy over the centralized chassis midplane (TCP)" and "Delivery model. Midplane notification is best-effort, not guaranteed. " . Why "TCP" is best effort here.

  6. Would Midplane message proxy be multiple threads, one for each LC? I assume zmqEmhanced producer is a separate process, so does proxy. How about zmqEnhancedconsumer, would it be part of orchagent, or a new process?

  7. "In steady state: enqueue decoded updates and wake the host consumer's event loop; deliver snapshot first, then deltas." What is "snapshot" referring here? "full database sync"?

  8. " coalesce on the pending midplane map to DELETE plus one merged SET nh=E." why do you keep delete event? We could merge to SET nh=E, a.k.a one route modify event.

  9. For full database sync, do you expect it would be used in OIR case only? since it is a big hammer. For OIR case, you may want to delay ports up after you finish snapshot delivery, to avoid other LC starting to use ports on this LC while it is not ready.

  10. For 8.4, can you add an analysis on the full database sync's impacts. For example, 8.4.2, consumer is slow to response. Adding a full database sync would make this problem even worse, right?

11.For zmqEnhanced consumer, you have a note "one per IPC -carried table"? What does it mean? how does it impact to orchagent?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants