diff --git a/doc/modular-chassis-voq/centralized_ipc_hld.md b/doc/modular-chassis-voq/centralized_ipc_hld.md new file mode 100644 index 00000000000..5cc0aacdb6e --- /dev/null +++ b/doc/modular-chassis-voq/centralized_ipc_hld.md @@ -0,0 +1,556 @@ +# Zmq + Redis based Inter-Process Communication (IPC) + +**Authors**: +Pradeep Revanniah — Cisco +Amit Grover — Cisco + +**Date:** 06/10/2026 + +--- + +## Table of Contents + +- [1. Definitions/Abbreviations](#1-definitionsabbreviations) +- [2. Revision](#2-revision) +- [3. Problem statement](#3-problem-statement) + - [3.1 Problem](#31-problem) + - [3.2 Solution](#32-solution) +- [4. Scope](#4-scope) +- [5. Assumptions](#5-assumptions) +- [6. Architecture](#6-architecture) + - [Midplane message proxy](#midplane-message-proxy) + - [zmqEnhanced producer](#zqmenhanced-producer) + - [zmqEnhanced consumer](#zqmenhanced-consumer) +- [7. Midplane message format](#7-midplane-message-format) + - [7.1 Frame Envelope](#71-frame-envelope) + - [7.2 Single-Entry Frame](#72-single-entry-frame) + - [7.3 Batched-Entry Frame](#73-batched-entry-frame) + - [7.4 Subscription Frames](#74-subscription-frames) +- [8. Operational IPC flows](#8-operational-ipc-flows) + - [8.1 Steady-state Update (Single Operation)](#81-steady-state-update-single-operation) + - [8.2 Batched Update](#82-batched-update) + - [8.3 Full database sync (slow-joiner mitigation)](#83-full-database-sync-slow-joiner-mitigation) + - [8.4 Failure and Recovery](#84-failure-and-recovery) + - [8.4.1 Proxy restart on Supervisor](#841-proxy-restart-on-supervisor) + - [8.4.2 Consumer back-pressure (message drop)](#842-consumer-back-pressure-message-drop) + - [8.4.3 No subscribers yet](#843-no-subscribers-yet) + - [8.4.4 zmqEnhanced consumer disconnect](#844-zqmenhanced-consumer-disconnect) + - [8.4.5 Producer back-pressure (send queue full)](#845-producer-back-pressure-send-queue-full) +- [9. IPC table set and sizing parameters](#9-ipc-table-set-and-sizing-parameters) +- [10. Restrictions and limitations](#10-restrictions-and-limitations) +- [11. Performance characterization](#11-performance-characterization) + - [11.1 Methodology](#111-methodology) + - [11.2 Convergence time](#112-convergence-time) + - [11.2.1 Redis-only approach](#1121-redis-only-approach) + - [11.2.2 Zmq + Redis based approach](#1122-zmq-redis-based-approach) + - [11.3 Comparison and design implications](#113-comparison-and-design-implications) +- [12. Related documents](#12-related-documents) + +--- + + + +## 1. Definitions/Abbreviations + +**Table 1: Definitions and abbreviations** + + +| Term | Definition | +| ---------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | +| zmqEnhanced IPC | Extension to producer and consumer state tables for **IPC-carried tables**: Redis write and midplane publish on the producer side; **full database sync** and midplane deltas on the consumer side. | +| zmqEnhanced producer | Enhancement to `ProducerStateTable` on the Supervisor for IPC-carried tables. Each operation performs a **zmqEnhanced IPC send**. No change to existing publish API. | +| zmqEnhanced consumer | Enhancement to `ConsumerStateTable` on the Linecardand for IPC-carried tables. Connects to the message proxy via miaplane, demultiplexes frames by `(database, table)`, One instance per IPC-carried table per NPU in **orchagent**. | +| Midplane | Control-plane interconnect between Supervisor and line cards in a centralized chassis. | +| Message proxy | XPUB/XSUB **Hub-and-Spoke** broker on the Supervisor. Stateless forwarder — ingests producer frames on the frontend and fans each frame to all connected consumers on the backend ([§6](#6-architecture)). | +| Zmq + Redis IPC fabric | Communication fabric for **platforms using this transport**: using **Hub-and-Spoke** proxy, **zmqEnhanced producers** on the producer side, and per-table **zmqEnhanced consumers** on the consumer side (one per NPU). | +| Central application database | Centralized Redis database on the Supervisor — **source of truth** for IPC-carried tables. Every zmqEnhanced IPC operation writes here first; line cards reconcile through **full database sync** when midplane notifications are missed ([§8.4](#84-failure-and-recovery)). | +| Full database sync | Read database snapshot for IPC-carried table from the **central application database**. Triggered at zmqEnhanced consumer start, reconnect, and consumer restart ([§8.3](#83-full-database-sync-slow-joiner-mitigation), [§8.4](#84-failure-and-recovery)). Live midplane deltas follow once sync completes. | +| Per-key coalescing | Within one buffered flush window, repeated operations on the same key collapse; Redis still executes every queued script ([§8.2](#82-batched-update)). | +| ProducerStateTable (PST) | Standard SONiC producer state table — `set` / `del` / `flush` with Redis scripted writes. Extended as **zmqEnhanced producer** for IPC-carried tables ([§6](#6-architecture)). | +| ConsumerStateTable (CST) | Standard SONiC consumer state table — drains pending keys via Redis pub/sub. Supports at most one consumer per table channel; does not scale to multi-NPU fan-out ([§3.1](#31-problem)). | +| Supervisor | Supervisor card in a centralized chassis; hosts the **central application database** and **midplane message proxy**. | +| LC (line card) | Line card in a centralized chassis; hosts per-NPU **orchagent** (**zmqEnhanced consumer** + host consumer) and **syncd**. | +| routeCheck | SONiC background consistency check comparing line-card programmed state against the **central application database**. Flags drift after midplane message loss; recovery triggers **full database sync** via service restart ([§8.4](#84-failure-and-recovery)). | + + +--- + + + +## 2. Revision + +**Table 2: Document revision history** + + +| Rev | Date | Author | Change Description | +| --- | ---------- | ---------------------------------------------- | ------------------------------- | +| 1.0 | 06/10/2026 | Pradeep Revanniah (Cisco), Amit Grover (Cisco) | Initial Zmq + Redis IPC design. | + + +--- + + + +## 3. Problem statement + + + +### 3.1 Problem + +In a **centralized chassis**, a control-plane **producer** on the Supervisor (for example, **fpmsyncd** writing `ROUTE_TABLE`) must deliver table updates to **many consumers** — one **orchagent** per NPU namespace on line cards. SONiC today offers two Redis-based notification paths for this class of update; **neither scales** on a centralized chassis. + +The `ProducerStateTable` **/** `ConsumerStateTable` **(PST/CST)** path is designed for **one publisher and at most one consumer**. The publisher inserts modified keys into a shared Redis **key set**, updates a **shadow table**, and publishes to a single table channel. The consumer wakes on that channel, **pops** keys from the key set, and applies updates. When a **second consumer** wakes on the same channel, the key set has already been **drained** by the first — it finds nothing to process. On a centralized chassis with multiple per-NPU orchagents, standard PST/CST **cannot fan out** the same update to every consumer. + +**Example:** A route published from **fpmsyncd** must reach every line-card **Route Orch** that programs the prefix locally. With standard PST/CST, only one orchagent reliably consumes the pending key set; others miss the notification cycle. + +The alternative — remote subscribers on **Redis keyspace notifications** with **PSUBSCRIBE** (pattern subscribe per table entry) — avoids the empty key-set problem but **does not scale**. Subscriber count grows with **(namespaces × tables × line cards)**. As subscriber count rises, measured **Redis publish throughput degrades**. Keyspace notifications are **not durably queued**; under load, **slow receivers** **application-level event loss**, with consistency depending on background checks (e.g. **routeCheck**) and **full database sync** for recovery. + +**Requirement** — Need 1→N fan-out, **full database sync** from the **central application database** at zmqEnhanced consumer construction, route-scale performance, and midplane notification without per-NPU **PSUBSCRIBE**. + +### 3.2 Solution + +This HLD defines the **Zmq + Redis IPC fabric** for a **centralized chassis** that: + +1. **Extends PST/CST for 1→N fan-out** — `ProducerStateTable` keeps the existing `set` / `del` / `flush` API and Redis write path; for IPC-carried tables it adds a **zmqEnhanced IPC send** (Redis write plus midplane publish). Each line card uses a **zmqEnhanced consumer** — a `ConsumerStateTable` subclass that preserves the `pops()` contract while receiving live midplane deltas instead of competing for the shared Redis key set. +2. **Decouples delivery from Redis pub/sub fan-in** — a **midplane message proxy** (ZMQ XPUB/XSUB **Hub-and-Spoke** broker) on the control-plane interconnect fans each published frame to all connected zmqEnhanced consumers with **one forwarding cost per frame**, without adding a remote **PSUBSCRIBE** session per NPU per table. +3. **Keeps Redis authoritative** — every zmqEnhanced IPC operation writes to the **central application database** first; zmqEnhanced consumers perform **full database sync** from that database at construction, after midplane reconnect, and when rebuilt after service restart, then drain midplane deltas — so notification loss on the midplane path does not replace Redis as the source of truth. Missed notifications are surfaced by **routeCheck** and reconciled through service restart and full database sync ([§8.4](#84-failure-and-recovery)). + +Implemented in **sonic-swss-common**; IPC-carried table set only ([§9](#9-ipc-table-set-and-sizing-parameters)); other paths unchanged (including **ProducerTable** / **ConsumerTable** 1:1 RPC). + +--- + + + +## 4. Scope + +This HLD covers: + +1. The **Zmq + Redis IPC fabric** — midplane XPUB/XSUB proxy, zmqEnhanced producer, and zmqEnhanced consumer ([§6](#6-architecture)). +2. **Midplane message format** — logical frame envelope and payload layouts ([§7](#7-midplane-message-format)). +3. **Operational flows** — steady state, batched updates, full database sync, and failure recovery ([§8](#8-operational-ipc-flows)). +4. The coordinated **IPC-carried table set** and transport sizing parameters ([§9](#9-ipc-table-set-and-sizing-parameters)). +5. **Design-validation performance** characterization motivating the fabric ([§11](#11-performance-characterization)). + +**Delivery model.** Midplane notification is **best-effort**, not guaranteed. Every producer operation still writes to the **central application database** first; any message loss on the midplane path must be recovered through **full database sync** ([§8.3](#83-full-database-sync-slow-joiner-mitigation), [§8.4](#84-failure-and-recovery)). + +Out of scope: + +- Application-level orchagent programming logic beyond the producer/consumer drain contract. + +--- + + + +## 5. Assumptions + +This design assumes the following: + +1. **Reachable IPC transport** — Producers on the Supervisor and zmqEnhanced consumers on line cards SHALL reach the same XPUB/XSUB proxy over the centralized chassis midplane (TCP). +2. **Authoritative Redis** — IPC-carried tables live in the **central application database** (Redis on the Supervisor). The producer **writes each update to that database**; the **zmqEnhanced IPC path** delivers the same changes to zmqEnhanced consumers over the **Zmq midplane**. Redis remains the **source of truth**; consumers reconcile through **full database sync** at construction, after reconnect, and when rebuilt after service restart; the midplane carries notification only. +3. **One proxy per chassis** — One midplane message proxy serves each centralized chassis on the Supervisor. Publishers connect to its frontend and zmqEnhanced consumers to its backend ([midplane message proxy](#midplane-message-proxy)). + +--- + + + +## 6. Architecture + +Today, SONiC distributes table updates through PST/CST over Redis: the producer writes to Redis and publishes a notification; the consumer drains pending entries. On a centralized chassis, multiple per-NPU consumers must receive the same update; standard PST/CST and remote Redis pub/sub do not scale for that 1→N fan-out ([§3.1](#31-problem)). The fabric below extends the existing producer/consumer interface for the IPC-carried table set ([§8](#8-operational-ipc-flows)). + +The Zmq + Redis IPC fabric is **not** a new application or container. It is a **plug-and-play extension** of the existing producer/consumer state-table mechanism in the shared SONiC library layer, comprising **three cooperating components** — midplane message proxy, zmqEnhanced producer, and zmqEnhanced consumer. + + +| Side | What changes | What stays the same | +| ------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------- | +| **Producer** | For IPC-carried tables, each publish performs write to the **central application database** and notification on the midplane ([zmqEnhanced producer](#zqmenhanced-producer)). | Application daemons keep the same publish contract they use today. | +| **Consumer** | For IPC-carried tables, the consumer performs a **full database sync** from the **central application database**, then drains **live midplane deltas** ([zmqEnhanced consumer](#zqmenhanced-consumer)). | The host consumer's event loop and per-table handler wiring are unchanged. | + + +Redis remains the **authoritative store** and the source for full database sync on every zmqEnhanced IPC operation. Tables outside the IPC-carried set continue to use the **existing Redis pub/sub interface**. The diagram below describe the **centralized chassis deployment**: Supervisor publisher (**fpmsyncd**), line-card **orchagent** (per NPU namespace), and the midplane message proxy on the Supervisor. + +Figure 1 — End-to-end architecture: fpmsyncd publishes on the Supervisor; per-NPU zmqEnhanced consumer on line cards feeds orchagent. + +**Figure 1: End-to-end architecture — Zmq + Redis IPC fabric (Hub-and-Spoke)** + + +| Step | Figure 1 | From → To | What happens | +| ----- | --------------------- | ---------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------- | +| **1** | fpmsyncd → Producer | Application daemon → producer state table | A table update is issued on the Supervisor. | +| **2** | zmqEnhanced IPC send | Producer → Central application database **and** Midplane message proxy | The same update is written to Redis and published on the midplane in one producer action (parallel paths). | +| **3** | Hub-and-Spoke fan-out | Midplane message proxy → zmqEnhanced consumer (×N) | The proxy forwards each frame to every connected spoke; forwarding cost is per frame, not per subscriber. | +| **4** | orchagent | zmqEnhanced consumer → consumer → ASIC application database → syncd | The consumer applies updates to local forwarding state and programs the ASIC. | + + +**Full database sync:** when a zmqEnhanced consumer is created while publishers are already active, it reads the full table from the **central application database** first, delivers that snapshot to the host consumer, then drains live midplane deltas ([§8.3](#83-full-database-sync-slow-joiner-mitigation)). + +**Table 3: Component summary** + + +| Component | Location | Role | +| ---------------------- | ------------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Midplane message proxy | Supervisor | XPUB/XSUB hub on the centralized chassis midplane. Stateless forwarder — no parse, persist, or replay. | +| zmqEnhanced producer | Supervisor application daemons | Extends the existing producer state table: Redis write plus midplane publish for IPC-carried tables. | +| zmqEnhanced consumer | Line-card host consumer, per NPU (**orchagent**) | One per IPC-carried table. Receives midplane frames, **full database sync** from the central application database at construction and after reconnect, delivers updates to the host consumer. | +| Host-consumer wiring | Line-card **orchagent** | Selects zmqEnhanced consumer vs standard PST/CST consumer for each table based on runtime flag and line-card mode. | + + +--- + + + +#### Midplane message proxy + +The midplane message proxy is the **Hub-and-Spoke broker** for Zmq + Redis IPC traffic. + + +| Socket | Pattern | Connects | Role | +| ------------ | ------- | -------------------------------------- | ----------------------------------------------- | +| **Frontend** | XSUB | Producers (PUB) connect in | Ingests every published frame. | +| **Backend** | XPUB | zmqEnhanced consumers (SUB) connect in | Fans each ingested frame to all current spokes. | + + +**Properties** + +- **Stateless** — no application parsing, no persistence, no replay. A frame published while the proxy is down is dropped ([§8.4.1](#841-proxy-restart-on-supervisor)). +- **Location** — runs on the Supervisor; bind address is the centralized chassis midplane. Producers and consumers learn endpoints through platform database configuration. +- **Trivial filtering** — each zmqEnhanced consumer demultiplexes locally on `(database, table)` identity. +- **One per chassis** — one proxy per centralized chassis. +- **Lifecycle** — starts after the **central application database** is up and before chassis configuration is applied; systemd `Restart=always` on exit; stop/start recovery hook cycles `sonic.target` on the Supervisor and line cards so spokes reconcile via full database sync at consumer construction ([§8.4](#84-failure-and-recovery)). +- **Observability** — periodic stats (received / forwarded / dropped / subscriber count) and shutdown summary. + + + +#### zmqEnhanced producer + +The zmqEnhanced producer is the **existing producer state table**, extended internally for IPC-carried tables. Application daemons require **no publish-contract change**. + +**zmqEnhanced IPC send** — every IPC-carried operation performs both paths in one call: + +1. **Write to Redis** — same scripted write path as the non-IPC case; any Redis-only observer sees unchanged semantics. +2. **Publish to midplane** — the same row operation is serialized and sent to the proxy frontend. + +The table below summarizes the four operation classes. Detailed batching, flush, and per-key coalescing behaviour is in [§8.2](#82-batched-update). + + +| Operation | Redis path | Midplane path | +| ------------- | ------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------- | +| Single set | Scripted write (queued in buffered mode). | Buffered: merged per key until flush. Unbuffered: immediate single-entry frame. | +| Single delete | Scripted delete (queued in buffered mode). | Buffered: coalesced per key until flush. Unbuffered: immediate single-entry frame. | +| Batched set | Pipelined batch write. | One batched-entry frame for the full vector. | +| Flush | Drains the Redis pipeline to the **central application database**. | Emits one batched-entry frame with per-key coalescing on the midplane map ([§8.2](#82-batched-update)). | + + +**Back-pressure** — midplane publish is non-blocking. A full producer send queue drops the frame, increments a per-producer counter, and returns soft failure; the Redis write still succeeds. TCP may reconnect underneath, but dropped notifications are not replayed. **routeCheck** flags line-card drift; recovery is a service restart and full database sync from the central application database ([§8.4.5](#845-producer-back-pressure-send-queue-full), [§8.3](#83-full-database-sync-slow-joiner-mitigation)). + +**Resources** — one publish socket per `(database, table)`; send buffers sized for route-scale bursts; TCP keep-alive on dead peers. + +**Observability** — messages sent, messages dropped, connection state, reconnect. + +#### zmqEnhanced consumer + +The zmqEnhanced consumer replaces Redis pub/sub notification for IPC-carried tables while preserving the **same drain contract** the host consumer uses today. On a centralized chassis it runs inside line-card **orchagent**, one per IPC-carried table per NPU namespace. + +**Responsibilities** + +- Connect to the proxy backend; receive and deserialize midplane frames. +- Accept only frames matching this consumer's `(database, table)` pair. +- On construction: **full database sync** — read the full table from the **central application database**, stage as the first delivery batch, then process live midplane deltas ([§8.3](#83-full-database-sync-slow-joiner-mitigation)). +- In steady state: enqueue decoded updates and wake the host consumer's event loop; deliver **snapshot first, then deltas**. +- On disconnect: bounded back-off reconnect, re-subscribe, **full database sync** for lost messages ([§8.4.4](#844-zqmenhanced-consumer-disconnect)). +- On consumer back-pressure: receiver queue full — message dropped; **routeCheck** flags drift; service restart reconciles via full database sync ([§8.4.2](#842-consumer-back-pressure-message-drop)). +- Maintain telemetry: received, delivered, dropped, reconnect attempts, connection state. + +--- + + + +## 7. Midplane message format + +This section describes the **logical shape** of a frame published on the midplane message bus — the contract between the [zmqEnhanced producer](#zqmenhanced-producer) and the [zmqEnhanced consumer](#zqmenhanced-consumer). The *logical envelope* below is what the two sides agree on. + +### 7.1 Frame Envelope + +Every midplane frame carries a small envelope identifying the source of the update, followed by a payload section whose shape depends on whether the frame is a single entry or a batched entry. + +Figure 2 — Midplane frame envelope: database, table, frame kind (single or batch), and payload (set or delete). + +**Figure 2: Midplane frame envelope — database : table : frame kind : payload.** + +| Envelope field | Direction | Purpose | +| ------------------- | --------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Database identifier | Producer → Subscriber | Names the logical Redis database the update belongs to (the **central application database** on the Supervisor). Subscribers reject any frame whose database identifier does not match their own database connection. | +| Table identifier | Producer → Subscriber | Names the Redis table (one of the entries in [§9](#9-ipc-table-set-and-sizing-parameters)). Subscribers reject any frame whose table identifier does not match the table they own. | +| Frame kind | Producer → Subscriber | One of `single` or `batch`. Selects the payload layout below. | + + +The proxy itself does **not** read the envelope; it forwards every received frame to every currently connected spoke ([midplane message proxy](#midplane-message-proxy)). The `(database, table)` demultiplex is performed locally ([zmqEnhanced consumer](#zqmenhanced-consumer)). + +### 7.2 Single-Entry Frame + +A single-entry frame carries exactly one operation on exactly one key. + + +| Payload field | Purpose | +| ---------------- | -------------------------------------------------------------------------------------------------- | +| Operation | One of `set` or `delete`. | +| Key | The Redis table key the operation applies to (e.g. a route prefix, an interface name, a LAG name). | +| Field-value list | Present for `set`; the row contents. Absent or empty for `delete`. | + + +Single-entry frames are produced by the single-set and single-delete operations described in the [zmqEnhanced producer](#zqmenhanced-producer). + +### 7.3 Batched-Entry Frame + +A batched-entry frame carries one or more **set** or **delete** operations as a length-prefixed list. + + +| Payload field | Purpose | +| ------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Entry count | Number of entries in this batch. | +| Entry 1 .. N | A repeated `` tuple per entry. Each entry preserves the same shape it would have in a single-entry frame, minus the envelope. | + + +Batched-entry frames are produced by the batched-set operation and by the flush operation when buffered mode has accumulated entries ([zmqEnhanced producer](#zqmenhanced-producer), [§8.2](#82-batched-update)). + +A consumer that receives a batched-entry frame surfaces the entire batch in a single delivery to the host consumer's event loop ([§8.2](#82-batched-update)). + +### 7.4 Subscription Frames + +Distinct from the data frames described above, ZeroMQ's XPUB/XSUB pattern also propagates **subscription frames** in the reverse direction (backend → frontend). These frames are produced by the underlying ZeroMQ library when a subscriber connects to the proxy's backend, and they are forwarded by the proxy back to its frontend so that publishers can become aware that at least one subscriber is attached. + +In the Zmq + Redis IPC fabric: + +- Per-table filtering is performed locally. +- Producers do not consume subscription frames for routing decisions; the proxy simply uses subscription state to track the current spoke count, which is exposed via the proxy stats line ([midplane message proxy](#midplane-message-proxy)). + +Subscription frames are mentioned here only for completeness; an integrator implementing a new subscriber does not need to construct them explicitly. + +--- + + + +## 8. Operational IPC flows + +This section describes end-to-end behaviour on a **centralized chassis** (Supervisor publishers, line-card consumers, ASIC programming on line cards). + +### 8.1 Steady-state Update (Single Operation) + +Figure 3 — Steady-state ladder: application daemon calls ProducerStateTable set/del; PST Redis write and zmqEnhanced IPC send; midplane message proxy → zmqEnhanced consumer → orchagent → ASIC. + +**Figure 3: Steady-state ladder — single IPC-carried table update (Hub-and-Spoke).** + +At construction, PST selects the zmqEnhanced path when the table is IPC-carried ([§9](#9-ipc-table-set-and-sizing-parameters)). Application daemons use the **existing** `ProducerStateTable` **API** (`set` / `del` / `flush`) — no publish-contract change. On each operation, PST performs the standard Redis write; for IPC-carried tables it also performs a **zmqEnhanced IPC send** to the midplane message proxy in the same call. The proxy forwards the frame to all connected spokes. The host consumer drains the queue and programs the ASIC. + +**Invariants.** + +- Redis remains the central store on every operation; midplane publish is best-effort and never gates the Redis write. +- Redis write and midplane publish happen in the same producer call. +- Midplane publish is non-blocking; a full producer send queue drops the frame and increments a per-producer counter ([§8.4.5](#845-producer-back-pressure-send-queue-full)). +- Per-table demultiplexing is performed locally based in filter. + + + +### 8.2 Batched Update + +The fabric supports batched delivery in two ways: + +1. **Explicit batched invocation** — the application passes a vector of updates in one producer call. Redis and the midplane each receive one aligned batch. +2. **Buffered accumulation** — many single-key updates accumulate in memory (typical BGP route churn on the Supervisor) and flush together. This is the dominant mode for `ROUTE_TABLE`. + + +| Mode | How triggered | Redis at flush | Midplane at flush | +| --------------------------- | ----------------------------------------------- | --------------------------------------------------------------------- | ------------------------------------------ | +| Explicit batched invocation | Producer called with a vector of updates | One pipelined script batch | One frame carrying the full vector | +| Buffered accumulation | Many single-key `set` / `del` calls, then flush | One pipeline flush executing **N** scripts (one per application call) | One frame carrying **M** coalesced entries | +| Per-key coalescing | Automatic on the pending midplane map | No coalescing — all **N** scripts still queued | Collapses repeated ops on the same key | + + +**Explicit vector.** One `set(vector)` call writes a pipelined Redis batch and sends one midplane frame. The zmqEnhanced consumer delivers the full batch in a single `pops()`. + +**Buffered routes.** `ROUTE_TABLE` PST runs in buffered mode: each `set` / `del` queues a Redis script and updates the pending midplane map. `flush()` is triggered when pipeline capacity is reached. On flush, PST performs a paired **Redis write** and **zmqEnhanced IPC send** ([§7.3](#73-batched-entry-frame)). + +**Per-key coalescing (midplane only).** Within one flush window, repeated operations on the same key collapse on the pending midplane map — merged `set` fields (last value wins per field), with `delete` clearing prior sets and preserving withdraw-then-re-add order when a `set` follows. Redis still executes every queued script. + +Figure 4 — Per-key coalescing: six buffered calls on 10.0.0.0/24 (set A/B/C, delete, set D/E); flush collapses the pending midplane map to DELETE plus SET nh=E. + +**Figure 4: Per-key coalescing — accumulate, flush, outcome (midplane only).** Six buffered calls on route key `10.0.0.0/24` — `set(nexthop=A/B/C)`, `delete`, `set(nexthop=D)`, `set(nexthop=E)` — coalesce on the pending midplane map to **DELETE** plus one merged **SET nh=E**. + +**Line card.** The zmqEnhanced consumer deserializes one frame, enqueues all entries, and wakes orchagent once. The host consumer drains the batch in one iteration. + +### 8.3 Full database sync (slow-joiner mitigation) + +Figure 5 — Full database sync ladder: consumer joins proxy and reads hub snapshot; PST queues live updates locally until snapshot complete; pops() snapshot batch, then PST sends queued deltas and pops() drains them. + +**Figure 5: Full database sync ladder — join, snapshot + PST hold, send queue + drain.** + +When the host consumer registers for an IPC-carried table, the zmqEnhanced consumer connects to the proxy backend, then reads the full table from the **central application database** as a one-shot snapshot staged for the first `pops()`. + +While the snapshot read is in progress, live updates at the zmqEnhanced PST accumulate in a **local pending queue on the Supervisor** — no midplane publish until the snapshot completes. When the snapshot read completes, the first `pops()` returns the snapshot batch. PST then sends held updates via **zmqEnhanced IPC send**; a subsequent `pops()` returns the queued live deltas. + +### 8.4 Failure and Recovery + +[Figure 6](#84-failure-and-recovery) shows the three saturation queues on the steady-state path from **ProducerStateTable** on the Supervisor to **orchagent** on a line card. Each numbered queue sits on the connector between stages: **①** PUB send queue, **②** XPUB fan-out (one queue per subscriber — queue1 … queue-n), and **③** received-op buffer before **orchagent** drains. When a marked queue saturates, midplane frames may be dropped while the **central application database** write on the Supervisor still completes. **①** maps to producer send-queue saturation ([§8.4.5](#845-producer-back-pressure-send-queue-full)), **②** to proxy XPUB fan-out with no subscribers or blocked send ([§8.4.3](#843-no-subscribers-yet)), and **③** to consumer receive-queue saturation ([§8.4.2](#842-consumer-back-pressure-message-drop)). + +Figure 6 — Vertical queue stack: ProducerStateTable, Midplane message proxy, zmqEnhanced consumer, orchagent; saturation queues ① PUB, ② XPUB fan-out (queue1 … queue-n), ③ received-op on the connectors between stages. + +**Figure 6: Saturation queues on the IPC stack (① PUB · ② XPUB fan-out · ③ received-op).** + +Five failure classes can interrupt midplane delivery. Each is detected by a different actor and increments a drop or restart counter. The central application database on the Supervisor remains authoritative in every case; line cards recover through **full database sync** ([§8.3](#83-full-database-sync-slow-joiner-mitigation), [§8.4](#84-failure-and-recovery)). + + +| Scenario | Detecting actor | On failure | Recovery | +| ---------------------------------------- | ------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Proxy restart on Supervisor | systemd (midplane proxy service `Restart=always`; recovery hook on stop/start) | Midplane notifications **not delivered** to zmqEnhanced consumers while the proxy is down; **Redis writes still succeed** on the Supervisor | systemd restarts the proxy and re-binds midplane endpoints; recovery handler cycles **sonic.target** on the Supervisor and all active line cards, restarting dependent services (including **orchagent**). Rebuilt zmqEnhanced consumers perform **full database sync** from the central application database at construction ([§8.3](#83-full-database-sync-slow-joiner-mitigation)); producers and subscribers reconnect to the proxy | +| Consumer back-pressure (message drop) | zmqEnhanced consumer (receiver queue full; message dropped) | Message **dropped** when receiver queue is full; per-consumer drop counter incremented | Subscriber does not retry the dropped frame. **routeCheck** flags programming drift on line cards; a **service restart** rebuilds zmqEnhanced consumers for **full database sync** from the central application database ([§8.3](#83-full-database-sync-slow-joiner-mitigation)) | +| Producer back-pressure (send queue full) | zmqEnhanced producer (send queue full; frame dropped) | Midplane frame **dropped**; per-producer drop counter incremented; **Redis write still succeeds** on the Supervisor | **routeCheck** flags programming drift on line cards; a **service restart** rebuilds zmqEnhanced consumers for **full database sync** from the central application database ([§8.3](#83-full-database-sync-slow-joiner-mitigation)) | +| No subscribers yet | Midplane message proxy | Frame dropped when no SUB socket is connected; drop counter visible in periodic stats | When subscriber connects, line cards rely on full database sync at construction ([§8.3](#83-full-database-sync-slow-joiner-mitigation)) | +| zmqEnhanced consumer disconnect | zmqEnhanced consumer | Frames published while disconnected are **lost** | Bounded back-off reconnect, re-subscribe, **full database sync** for lost messages ([§8.3](#83-full-database-sync-slow-joiner-mitigation)) | + + + + +#### 8.4.1 Proxy restart on Supervisor + +While the midplane proxy is down, notifications are not delivered to zmqEnhanced consumers but **Redis writes still succeed** on the Supervisor. A systemd recovery hook restarts the proxy and cycles `sonic.target` on the Supervisor and line cards; rebuilt consumers perform full database sync from the central application database ([§8.3](#83-full-database-sync-slow-joiner-mitigation)). + +#### 8.4.2 Consumer back-pressure (message drop) + +When the receiver queue is full, the message is **dropped** and the per-consumer drop counter is incremented. The subscriber does not retry. **routeCheck** flags programming drift; a service restart rebuilds zmqEnhanced consumers for full database sync ([§8.3](#83-full-database-sync-slow-joiner-mitigation)). + +#### 8.4.3 No subscribers yet + +The proxy drops frames while no SUB socket is connected. When a subscriber connects, line cards rely on full database sync at zmqEnhanced consumer construction ([§8.3](#83-full-database-sync-slow-joiner-mitigation)). + +#### 8.4.4 zmqEnhanced consumer disconnect + +Frames published while the zmqEnhanced consumer is disconnected are **lost**. Recovery is bounded back-off reconnect, re-subscribe, and full database sync for lost messages ([§8.3](#83-full-database-sync-slow-joiner-mitigation)). + +#### 8.4.5 Producer back-pressure (send queue full) + +When the producer send queue is full, the midplane frame is **dropped** and the per-producer drop counter is incremented; the **Redis write still succeeds**. **routeCheck** flags programming drift; a service restart rebuilds zmqEnhanced consumers for full database sync ([§8.3](#83-full-database-sync-slow-joiner-mitigation)). + +--- + + + +## 9. IPC table set and sizing parameters + +The Zmq + Redis IPC sizing is expressed through **coordinated parameters** in the producer, proxy, and consumer for route scale and chassis fan-out. The zmqEnhanced IPC path is **database-agnostic**: any Redis database may participate when its tables are added to the coordinated producer/consumer set on **both** sides. + +The tables below use the ZeroMQ midplane on a centralized chassis (central Supervisor application database). Additional tables may be added through a coordinated producer and consumer change. Queue placement and saturation behavior are summarized in [Figure 6](#84-failure-and-recovery). + +**IPC-carried tables:** `BUFFER_PG_TABLE`, `BUFFER_POOL_TABLE`, `BUFFER_PORT_EGRESS_PROFILE_LIST_TABLE`, `BUFFER_PORT_INGRESS_PROFILE_LIST_TABLE`, `BUFFER_PROFILE_TABLE`, `BUFFER_QUEUE_TABLE`, `COPP_TABLE`, `FABRIC_MONITOR_TABLE`, `FABRIC_PORT_TABLE`, `INTF_TABLE`, `LAG_MEMBER_TABLE`, `LAG_TABLE`, `NEIGH_TABLE`, `PORT_TABLE`, `ROUTE_TABLE`, `VRF_TABLE` + +**Table 4: Coordinated sizing parameters** + + +| Component | Sizing parameter | Design intent | +| ------------------------------------------- | ----------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| zmqEnhanced producer (buffered) | Redis pipeline depth (`COMMAND_MAX`) | Cap how many scripted Redis operations accumulate before a pipeline flush; route producers are sized for large burst windows | +| | Idle / timer flush interval | Time-based drain of buffered producer work when churn pauses | +| | Per-key coalescing (pending midplane map) | Collapse repeated operations on the same key within one flush window (Redis still executes one script per application call) ([§8.2](#82-batched-update)) | +| zmqEnhanced producer (transport) | PUB send high-water mark (`SNDHWM`) | Bound the producer socket send queue; saturation drops frames rather than blocking the application thread | +| | PUB socket send buffer (`SNDBUF`) | OS-level egress buffering on the publish socket | +| Midplane message proxy | XSUB receive high-water mark and receive buffer | Ingress buffering from all producers before the forward loop | +| | XPUB send high-water mark and send buffer | Egress buffering toward all line-card subscribers | +| | Context I/O thread count and max sockets | Forwarding capacity for chassis-scale fan-out | +| zmqEnhanced consumer (`ZmqEnhancedClient`) | SUB receive high-water mark (`RCVHWM`) | Socket-level burst absorption on the line card | +| | SUB socket receive buffer (`RCVBUF`) | OS-level ingress buffering on the subscribe socket | + + +--- + + + +## 10. Restrictions and limitations + + +| Item | Detail | +| ------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| IPC-carried table set | Fixed, coordinated producer/consumer list ([§9](#9-ipc-table-set-and-sizing-parameters)); tables outside the set use the existing PST/CST interface unchanged | +| Midplane delivery | Best-effort; Redis write always succeeds; dropped midplane frames are not replayed by the proxy | +| Full database sync scope | Consumer reconciles from the **central application database** through a full table read at construction, after midplane reconnect following disconnect, and when rebuilt after service restart ([§8.3](#83-full-database-sync-slow-joiner-mitigation), [§8.4](#84-failure-and-recovery)) | +| Proxy placement | One midplane message proxy per centralized chassis, on the Supervisor | +| Subscriber scale | One SUB socket and worker thread per (IPC-carried table × subscriber instance) | + + +--- + + + +## 11. Performance characterization + +**Disclaimer. All performance numbers in this section were collected from tests performed on Cisco hardware in a Controlled Cisco lab environment. They are representative design-validation measurements from pre-deployment validation, not a hard performance contract or a guarantee on other platforms. Specific platform details influence absolute values; the shape of the curves — and the qualitative conclusions drawn from them.** + +### 11.1 Methodology + +- **Setup.** A centralized chassis with one Supervisor and N line cards, deployed on Cisco hardware in a Cisco lab environment. Updates are produced on the Supervisor (route table is the dominant load source); zmqEnhanced consumers run inside per-NPU **orchagent** on line cards. +- **Configurations under test.** Two approaches are compared on the same hardware: + 1. **Redis-only approach** — every line-card consumer attaches to the Supervisor's **central application database** via a Redis pub/sub session. + 2. **Zmq + Redis based approach** — the same producers use the **Zmq + Redis IPC fabric** in this document: Redis remains the central store, but live notification fan-out moves to the Hub-and-Spoke midplane message proxy and **zmqEnhanced consumers** on line cards. +- **Workloads.** [§11.2](#112-convergence-time) measures **convergence time** — bulk insert of N routes and wall-clock time until each subscriber has all N routes. + + + +### 11.2 Convergence time + +The motivating performance question for a centralized chassis is route-scale **convergence under fan-out**: how long until every line-card **orchagent** has applied a bulk route insert from the Supervisor. [Figure 7](#112-convergence-time) plots the same convergence workload side by side — Redis-only on the left, Zmq + Redis on the right — with single-subscriber time and median time across 15 line-card subscribers at each route count. + +Figure 7 — Convergence time vs route count: Redis-only approach (left) shows subscriber time growing with route count and a widening gap between one subscriber and the median across 15 line-card subscribers at 100k routes (7.5 s vs 26.5 s). Zmq + Redis based approach (right) shows single and median times tracking together through 100k routes (9.6 s vs 9.2 s) — fan-out stays flat. + +**Figure 7: Convergence time vs route count** + +#### 11.2.1 Redis-only approach + +Bulk insert of N routes into the central route table on the Supervisor; measure wall-clock time until each consumer has all N routes. Reported as subscriber time in seconds. + +**Observations.** + +- Single-subscriber time grows roughly linearly with route count, from ~1.5 s at 10k routes to ~7.5 s at 100k routes — i.e. each additional 10k routes adds roughly 0.5–1 s of convergence delay. +- The 15-subscriber median is **substantially worse** at every route count, opening a 2–3× gap by 60k routes and a 3.5× gap at 100k routes (≈26.5 s vs ≈7.5 s). Crucially, the gap does **not** close at higher route counts; it widens. This is the defining symptom of the 1-producer → 1-consumer Redis fan-out: each additional consumer adds an independent Redis pub/sub session that competes for the same Supervisor Redis CPU and notification queue. + + + +#### 11.2.2 Zmq + Redis based approach + +Same convergence workload on the Zmq + Redis based approach. + +**Observations.** + +- Single-subscriber convergence grows with route count but stays bounded — ≈9.6 s at 100k routes on the measured hardware path. +- The 15-subscriber median **tracks the single-subscriber curve**. Fan-out cost does not multiply with line-card count. + + + +### 11.3 Comparison and design implications + +[Figure 7](#112-convergence-time) shows the convergence story: Redis-only fan-out degrades the 15-subscriber median to ≈26.5 s at 100k routes while Zmq + Redis keeps median near single-subscriber time (≈9.2 s). Qualitative comparison: + + +| Property | Redis-only approach | Zmq + Redis based approach | Implication | +| ---------------------------- | ---------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------ | +| Notification path | Redis pub/sub session per `(producer, consumer)` pair | One Hub-and-Spoke proxy frame per producer call, fanned out by ZeroMQ | Zmq + Redis IPC removes Supervisor Redis as the fan-out bottleneck. | +| Cost as N consumers grows | Per-consumer Redis sessions multiply; Supervisor Redis CPU and notification queue scale with N | Per-frame proxy cost is constant; ZeroMQ replicates one frame to all current spokes in one forwarding step | Zmq + Redis IPC offers **constant** throughput as line-card / NPU count rises. | +| Convergence at scale | 100k routes × 15 subscribers ≈ **26.5 s median** ([§11.2.1](#1121-redis-only-approach)) | 100k routes × 15 subscribers ≈ **9.2 s median** ([§11.2.2](#1122-zmq-redis-based-approach)) | Zmq + Redis IPC removes the multi-subscriber convergence penalty that defines the Redis-only baseline. | +| End-to-end hot-path delivery | Redis pub/sub round-trip; polling-style notification | One proxy hop on the midplane; sub-millisecond forwarding step | Zmq + Redis IPC is **sub-millisecond** on the wire portion of the path. | +| Central store state | Redis | Redis (unchanged) | Zmq + Redis IPC is **purely additive**; non-IPC consumers see identical Redis semantics. | + + +The qualitative claims that drive the design — **sub-millisecond delivery on the hot path** and **flat convergence under multi-subscriber fan-out** — are supported by [Figure 7](#112-convergence-time) and the comparison rows above. For chassis sizing, the key quantitative result is the flat ≈9 s 15-subscriber median at 100k routes ([§11.2.2](#1122-zmq-redis-based-approach)), which decouples fabric capacity from topology. + +--- + + + +## 12. Related documents + + +| Document | Link | +| ------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Centralized chassis baseline | [voq_chassis_hld.md](https://wwwin-github.cisco.com/whitebox/SONiC/blob/master/doc/centralized-chassis/voq_chassis_hld.md) | +| Centralized chassis routing HLD | [centralized_chassis_routing_hld.md](https://wwwin-github.cisco.com/whitebox/SONiC/blob/centralized-chassis-routing-hld/doc/centralized-chassis/centralized_chassis_routing_hld.md) | + + +--- + +*End of document.* \ No newline at end of file diff --git a/doc/modular-chassis-voq/img/centralized_ipc_flow_cold_start.png b/doc/modular-chassis-voq/img/centralized_ipc_flow_cold_start.png new file mode 100644 index 00000000000..158b67acf1b Binary files /dev/null and b/doc/modular-chassis-voq/img/centralized_ipc_flow_cold_start.png differ diff --git a/doc/modular-chassis-voq/img/centralized_ipc_flow_steady_state.png b/doc/modular-chassis-voq/img/centralized_ipc_flow_steady_state.png new file mode 100644 index 00000000000..768226e9116 Binary files /dev/null and b/doc/modular-chassis-voq/img/centralized_ipc_flow_steady_state.png differ diff --git a/doc/modular-chassis-voq/img/centralized_ipc_frame_envelope.png b/doc/modular-chassis-voq/img/centralized_ipc_frame_envelope.png new file mode 100644 index 00000000000..3c23b1b6531 Binary files /dev/null and b/doc/modular-chassis-voq/img/centralized_ipc_frame_envelope.png differ diff --git a/doc/modular-chassis-voq/img/centralized_ipc_message_flow.png b/doc/modular-chassis-voq/img/centralized_ipc_message_flow.png new file mode 100644 index 00000000000..7c07535a029 Binary files /dev/null and b/doc/modular-chassis-voq/img/centralized_ipc_message_flow.png differ diff --git a/doc/modular-chassis-voq/img/centralized_ipc_per_key_coalescing.png b/doc/modular-chassis-voq/img/centralized_ipc_per_key_coalescing.png new file mode 100644 index 00000000000..0f93e301f7d Binary files /dev/null and b/doc/modular-chassis-voq/img/centralized_ipc_per_key_coalescing.png differ diff --git a/doc/modular-chassis-voq/img/centralized_ipc_perf_analysis_composite.png b/doc/modular-chassis-voq/img/centralized_ipc_perf_analysis_composite.png new file mode 100644 index 00000000000..b7204b5a842 Binary files /dev/null and b/doc/modular-chassis-voq/img/centralized_ipc_perf_analysis_composite.png differ diff --git a/doc/modular-chassis-voq/img/centralized_ipc_perf_hybrid_throughput.png b/doc/modular-chassis-voq/img/centralized_ipc_perf_hybrid_throughput.png new file mode 100644 index 00000000000..6d01fe40c01 Binary files /dev/null and b/doc/modular-chassis-voq/img/centralized_ipc_perf_hybrid_throughput.png differ diff --git a/doc/modular-chassis-voq/img/centralized_ipc_perf_redis_convergence.png b/doc/modular-chassis-voq/img/centralized_ipc_perf_redis_convergence.png new file mode 100644 index 00000000000..f9daca1cc9a Binary files /dev/null and b/doc/modular-chassis-voq/img/centralized_ipc_perf_redis_convergence.png differ diff --git a/doc/modular-chassis-voq/img/centralized_ipc_queueing_stack.png b/doc/modular-chassis-voq/img/centralized_ipc_queueing_stack.png new file mode 100644 index 00000000000..f3eb6b24a7e Binary files /dev/null and b/doc/modular-chassis-voq/img/centralized_ipc_queueing_stack.png differ diff --git a/doc/modular-chassis-voq/img/figure_common.py b/doc/modular-chassis-voq/img/figure_common.py new file mode 100644 index 00000000000..792894ba55e --- /dev/null +++ b/doc/modular-chassis-voq/img/figure_common.py @@ -0,0 +1,113 @@ +"""Shared drawing helpers for §8 Operational IPC flow figures.""" + +from PIL import Image, ImageDraw, ImageFont + +W = 1536 +BG = (255, 255, 255) +GRID = (220, 220, 220) +TEXT = (30, 30, 30) +MUTED = (90, 90, 90) +ARROW = (60, 60, 60) + +# Figure 5 reference participants (hub → spoke) +HUB_PARTICIPANTS = [ + ("Application\ndaemon", (66, 133, 244)), + ("ProducerStateTable", (142, 68, 173), "(zmqEnhanced path for IPC-carried tables)"), + ("Central Application\nDatabase", (231, 76, 60)), + ("Midplane message\nproxy", (230, 126, 34)), + ("zmqEnhanced\nconsumer", (39, 174, 96), "(one per IPC-carried table)"), + ("orchagent", (52, 152, 219), "(host consumer)"), + ("ASIC", (127, 140, 141)), +] + +PST = ("ProducerStateTable", (142, 68, 173), "(zmqEnhanced path)") +DB = ("Central Application\nDatabase", (231, 76, 60)) +PROXY = ("Midplane message\nproxy", (230, 126, 34)) +CONSUMER = ("zmqEnhanced\nconsumer", (39, 174, 96), "(one per IPC-carried table)") +ORCH = ("orchagent", (52, 152, 219), "(host consumer)") + + +def load_font(size: int, bold: bool = False): + candidates = [ + "/usr/share/fonts/truetype/dejavu/DejaVuSans-Bold.ttf" if bold else "/usr/share/fonts/truetype/dejavu/DejaVuSans.ttf", + "/usr/share/fonts/truetype/liberation/LiberationSans-Bold.ttf" if bold else "/usr/share/fonts/truetype/liberation/LiberationSans-Regular.ttf", + ] + for path in candidates: + try: + return ImageFont.truetype(path, size) + except OSError: + continue + return ImageFont.load_default() + + +def wrap_center(draw, text, cx, y, font, fill=TEXT, line_gap=4): + lines = text.split("\n") + cy = y + for line in lines: + bbox = draw.textbbox((0, 0), line, font=font) + w = bbox[2] - bbox[0] + h = bbox[3] - bbox[1] + draw.text((cx - w / 2, cy), line, font=font, fill=fill) + cy += h + line_gap + return cy - y + + +def draw_title_block(draw, title, subtitle, title_font, sub_font): + bbox = draw.textbbox((0, 0), title, font=title_font) + draw.text(((W - (bbox[2] - bbox[0])) / 2, 24), title, font=title_font, fill=TEXT) + bbox = draw.textbbox((0, 0), subtitle, font=sub_font) + draw.text(((W - (bbox[2] - bbox[0])) / 2, 62), subtitle, font=sub_font, fill=MUTED) + + +def draw_participants(draw, participants, margin_x=48, top_y=110, header_h=92, width=W, img_h=918): + n = len(participants) + col_w = (width - 2 * margin_x) / n + xs = [margin_x + col_w * (i + 0.5) for i in range(n)] + head_font = load_font(15, bold=True) + subhead_font = load_font(12) + for i, p in enumerate(participants): + color = p[1] + x0 = margin_x + col_w * i + 6 + x1 = margin_x + col_w * (i + 1) - 6 + draw.rounded_rectangle([x0, top_y, x1, top_y + header_h], radius=10, fill=color + (40,), outline=color, width=2) + wrap_center(draw, p[0], xs[i], top_y + 14, head_font, fill=TEXT) + if len(p) > 2: + wrap_center(draw, p[2], xs[i], top_y + 52, subhead_font, fill=MUTED) + lifeline_top = top_y + header_h + 8 + lane_bottom = img_h - 36 + for x in xs: + for y in range(lifeline_top, lane_bottom, 10): + draw.line([(x, y), (x, min(y + 5, lane_bottom))], fill=GRID, width=2) + return xs, lifeline_top + + +def draw_arrow(draw, xs, y, src, dst, label, step_font, dashed=False): + x0, x1 = xs[src], xs[dst] + if dashed: + for x in range(int(min(x0, x1)), int(max(x0, x1)), 14): + x_end = min(x + 8, max(x0, x1)) + draw.line([(x, y), (x_end, y)], fill=ARROW, width=2) + else: + draw.line([(x0, y), (x1, y)], fill=ARROW, width=2) + if x1 > x0: + draw.polygon([(x1, y), (x1 - 10, y - 5), (x1 - 10, y + 5)], fill=ARROW) + else: + draw.polygon([(x1, y), (x1 + 10, y - 5), (x1 + 10, y + 5)], fill=ARROW) + wrap_center(draw, label, (x0 + x1) / 2, y - 28, step_font, fill=TEXT) + + +def draw_ladder(img_h, participants, steps, gaps, title, subtitle): + img = Image.new("RGB", (W, img_h), BG) + draw = ImageDraw.Draw(img) + title_font = load_font(28, bold=True) + sub_font = load_font(16) + step_font = load_font(14) + draw_title_block(draw, title, subtitle, title_font, sub_font) + xs, lifeline_top = draw_participants(draw, participants, top_y=110, header_h=92, img_h=img_h) + y = lifeline_top + 55 + for idx, step in enumerate(steps): + src, dst, label = step[0], step[1], step[2] + dashed = len(step) > 3 and step[3] + draw_arrow(draw, xs, y, src, dst, label, step_font, dashed=dashed) + y += gaps[idx] + return img diff --git a/doc/modular-chassis-voq/img/gen_frame_envelope_figure.py b/doc/modular-chassis-voq/img/gen_frame_envelope_figure.py new file mode 100644 index 00000000000..9d57a407cdd --- /dev/null +++ b/doc/modular-chassis-voq/img/gen_frame_envelope_figure.py @@ -0,0 +1,104 @@ +#!/usr/bin/env python3 +"""Generate §7.1 Frame Envelope figure — database : table : frame kind : payload.""" + +import os + +from PIL import Image, ImageDraw + +from figure_common import BG, MUTED, TEXT, load_font, wrap_center + +IMG_DIR = os.path.dirname(os.path.abspath(__file__)) +OUT = os.path.join(IMG_DIR, "centralized_ipc_frame_envelope.png") + +IMG_W = 1024 +IMG_H = 210 + +C_DB = (66, 133, 244) +C_TABLE = (142, 68, 173) +C_KIND = (230, 126, 34) +C_PAYLOAD = (39, 174, 96) +C_FRAME = (120, 120, 140) + + +def draw_title_block(draw, title, subtitle, title_font, sub_font, width): + bbox = draw.textbbox((0, 0), title, font=title_font) + draw.text(((width - (bbox[2] - bbox[0])) / 2, 12), title, font=title_font, fill=TEXT) + bbox = draw.textbbox((0, 0), subtitle, font=sub_font) + draw.text(((width - (bbox[2] - bbox[0])) / 2, 38), subtitle, font=sub_font, fill=MUTED) + + +def segment(draw, x, y, w, h, title, value, color, title_font, value_font): + draw.rounded_rectangle([x, y, x + w, y + h], radius=10, outline=color, width=2, fill=(255, 255, 255)) + # color accent bar + draw.rounded_rectangle([x + 2, y + 2, x + w - 2, y + 24], radius=8, fill=color + (48,)) + wrap_center(draw, title, x + w / 2, y + 5, title_font, fill=TEXT) + wrap_center(draw, value, x + w / 2, y + h // 2 + 6, value_font, fill=TEXT) + + +def colon(draw, x, y, font): + bbox = draw.textbbox((0, 0), ":", font=font) + tw = bbox[2] - bbox[0] + th = bbox[3] - bbox[1] + draw.text((x - tw / 2, y - th / 2), ":", font=font, fill=(100, 100, 100)) + + +def main(): + title_font = load_font(22, bold=True) + sub_font = load_font(13) + seg_title_font = load_font(12, bold=True) + value_font = load_font(15, bold=True) + colon_font = load_font(22, bold=True) + + img = Image.new("RGB", (IMG_W, IMG_H), BG) + draw = ImageDraw.Draw(img) + draw_title_block( + draw, + "Midplane Frame Envelope", + "Logical layout of every data frame on the midplane bus", + title_font, + sub_font, + IMG_W, + ) + + margin = 40 + frame_x = margin + frame_y = 68 + frame_w = IMG_W - 2 * margin + frame_h = 100 + draw.rounded_rectangle( + [frame_x, frame_y, frame_x + frame_w, frame_y + frame_h], + radius=12, + outline=C_FRAME, + width=2, + fill=(250, 251, 253), + ) + wrap_center(draw, "Midplane data frame", frame_x + frame_w / 2, frame_y + 10, seg_title_font, fill=MUTED) + + inner_y = frame_y + 28 + inner_h = frame_h - 34 + gap = 16 + n_cols = 4 + total_gap = gap * (n_cols - 1) + 28 # colons + padding + col_w = (frame_w - total_gap) // n_cols + x = frame_x + 14 + + cols = [ + ("Database", "APPL_DB", C_DB), + ("Table", "ROUTE_TABLE", C_TABLE), + ("Frame kind", "single | batch", C_KIND), + ("Payload", "set / delete …", C_PAYLOAD), + ] + + for i, (title, value, color) in enumerate(cols): + segment(draw, x, inner_y, col_w, inner_h, title, value, color, seg_title_font, value_font) + x += col_w + if i < n_cols - 1: + colon(draw, x + gap / 2, inner_y + inner_h / 2, colon_font) + x += gap + + img.save(OUT, "PNG") + print("Wrote", OUT, f"({IMG_W}x{IMG_H})") + + +if __name__ == "__main__": + main() diff --git a/doc/modular-chassis-voq/img/gen_ipc_queueing_stack.py b/doc/modular-chassis-voq/img/gen_ipc_queueing_stack.py new file mode 100644 index 00000000000..26793ba8339 --- /dev/null +++ b/doc/modular-chassis-voq/img/gen_ipc_queueing_stack.py @@ -0,0 +1,146 @@ +#!/usr/bin/env python3 +"""Figure 6 — queues sit beside the vertical connector (socket on the wire).""" + +import os + +from PIL import Image, ImageDraw + +from figure_common import BG, MUTED, TEXT, W, draw_title_block, load_font, wrap_center + +IMG_DIR = os.path.dirname(os.path.abspath(__file__)) +OUT = os.path.join(IMG_DIR, "centralized_ipc_queueing_stack.png") + +C_PST = (142, 68, 173) +C_PROXY = (230, 126, 34) +C_CONSUMER = (39, 174, 96) +C_ORCH = (52, 152, 219) +C_ARROW = (60, 60, 60) +Q_ZMQ = (230, 126, 34) +Q_INPROC = (142, 68, 173) +C_SAT_BG = (241, 245, 249) +C_SAT_BORDER = (203, 213, 225) +C_SAT_TEXT = (71, 85, 105) + + +def fig1_box(draw, x, y, w, h, title, color, font): + draw.rounded_rectangle([x, y, x + w, y + h], radius=10, outline=color, width=2, fill=color + (40,)) + bbox = draw.textbbox((0, 0), title, font=font) + th = bbox[3] - bbox[1] + draw.text((x + (w - (bbox[2] - bbox[0])) / 2, y + (h - th) / 2 - 1), title, font=font, fill=TEXT) + + +def sat_badge(draw, x, y, num, font): + label = str(num) + tw = draw.textbbox((0, 0), label, font=font)[2] + r = 11 + draw.ellipse([x - r, y - r, x + r, y + r], fill=C_SAT_BG, outline=C_SAT_BORDER, width=1) + draw.text((x - tw / 2, y - 6), label, font=font, fill=C_SAT_TEXT) + + +def draw_socket_queue(draw, x, y, w, h, color, n_slabs=5, sat=None, sat_f=None): + """Vertical queue stack — sits beside the connector arrow.""" + draw.rounded_rectangle([x, y, x + w, y + h], radius=6, outline=color, width=2, fill=(252, 253, 255)) + pad = 4 + slab_h = max(4, (h - pad * 2 - (n_slabs - 1) * 2) // n_slabs) + sy = y + h - pad - slab_h + for i in range(n_slabs): + t = 45 + i * 22 + fill = tuple(min(255, int(c * (1 - t / 255) + 255 * (t / 255))) for c in color) + draw.rounded_rectangle([x + pad, sy, x + w - pad, sy + slab_h], radius=2, outline=color, width=1, fill=fill) + sy -= slab_h + 2 + if sat is not None: + sat_badge(draw, x + w + 2, y + 4, sat, sat_f) + + +def arrow_v(draw, x, y0, y1): + draw.line([(x, y0 + 2), (x, y1 - 10)], fill=C_ARROW, width=2) + draw.polygon([(x, y1 - 4), (x - 5, y1 - 12), (x + 5, y1 - 12)], fill=C_ARROW) + + +def connector_with_queue(draw, lane_x, y, conn_h, color, sat, sat_f, q_w=54, q_h=48): + """Vertical arrow with single queue beside it — socket on the wire.""" + y1 = y + conn_h + arrow_v(draw, lane_x, y, y1) + qx = lane_x + 20 + qy = y + (conn_h - q_h) // 2 + draw_socket_queue(draw, qx, qy, q_w, q_h, color, n_slabs=5, sat=sat, sat_f=sat_f) + return y1 + + +def connector_with_fanout_queues(draw, lane_x, y, conn_h, color, sat, sat_f, label_f): + """XPUB fan-out — queue1, queue2, …, queue-n beside the connector.""" + y1 = y + conn_h + arrow_v(draw, lane_x, y, y1) + + q_w, q_h = 42, 38 + gap = 6 + ell_w = 14 + labels = ["queue1", "queue2", "queue-n"] + n_queues = 3 + group_w = n_queues * q_w + (n_queues - 1) * gap + ell_w + gap + qx = lane_x + 20 + qy = y + 6 + + sat_badge(draw, qx + group_w - 2, qy - 2, sat, sat_f) + + x = qx + for i, lbl in enumerate(labels): + if i == 2: + draw.text((x + 2, qy + q_h // 2 - 6), "…", font=label_f, fill=MUTED) + x += ell_w + gap + draw_socket_queue(draw, x, qy, q_w, q_h, color, n_slabs=4) + tw = draw.textbbox((0, 0), lbl, font=label_f)[2] + draw.text((x + (q_w - tw) / 2, qy + q_h + 3), lbl, font=label_f, fill=MUTED) + x += q_w + gap + + return y1 + + +def main(): + lane_x = W // 2 + box_w = 380 + box_h = 56 + box_x = lane_x - box_w // 2 + conn = 62 + q_w, q_h = 54, 48 + + title_f = load_font(26, bold=True) + sub_f = load_font(14) + box_f = load_font(14, bold=True) + cap_f = load_font(12) + sat_f = load_font(10, bold=True) + q_label_f = load_font(9) + + img = Image.new("RGB", (W, 580), BG) + draw = ImageDraw.Draw(img) + + draw_title_block(draw, "Queueing Across the IPC Stack", + "Saturation points on the steady-state path", title_f, sub_f) + + y = 92 + + fig1_box(draw, box_x, y, box_w, box_h, "ProducerStateTable", C_PST, box_f) + y += box_h + y = connector_with_queue(draw, lane_x, y, conn, Q_ZMQ, 1, sat_f, q_w, q_h) + + fig1_box(draw, box_x, y, box_w, box_h, "Midplane message proxy", C_PROXY, box_f) + y += box_h + y = connector_with_fanout_queues(draw, lane_x, y, conn, Q_ZMQ, 2, sat_f, q_label_f) + + fig1_box(draw, box_x, y, box_w, box_h, "zmqEnhanced consumer", C_CONSUMER, box_f) + y += box_h + y = connector_with_queue(draw, lane_x, y, conn, Q_INPROC, 3, sat_f, q_w, q_h) + + fig1_box(draw, box_x, y, box_w, box_h, "orchagent", C_ORCH, box_f) + y += box_h + 24 + + wrap_center(draw, "Figure — Saturation queues ① PUB ② XPUB ③ received-op", + lane_x, y, cap_f, fill=MUTED) + + img = img.crop((0, 0, W, y + 24)) + img.save(OUT, "PNG") + print(f"Wrote {OUT} ({img.size[0]}x{img.size[1]})") + + +if __name__ == "__main__": + main() diff --git a/doc/modular-chassis-voq/img/gen_perf_composite.py b/doc/modular-chassis-voq/img/gen_perf_composite.py new file mode 100644 index 00000000000..0009e645131 --- /dev/null +++ b/doc/modular-chassis-voq/img/gen_perf_composite.py @@ -0,0 +1,86 @@ +#!/usr/bin/env python3 +"""Generate side-by-side performance composite (slide 4 style) for local review.""" + +import os +from PIL import Image, ImageDraw, ImageFont + +IMG_DIR = os.path.dirname(os.path.abspath(__file__)) +LEFT_SRC = os.path.join(IMG_DIR, "centralized_ipc_perf_redis_convergence.png") +RIGHT_SRC = os.path.join(IMG_DIR, "centralized_ipc_perf_hybrid_throughput.png") +OUT = os.path.join(IMG_DIR, "centralized_ipc_perf_analysis_composite.png") + +BG = (255, 255, 255) +TITLE_BG = (0, 48, 87) # slide-like banner +TITLE_FG = (255, 255, 255) +SUBTITLE_FG = (30, 30, 30) +RULE = (200, 200, 200) +PAD = 32 +GAP = 24 +BANNER_H = 72 +SUBTITLE_H = 44 + + +def load_font(size: int, bold: bool = False): + candidates = [ + "/usr/share/fonts/truetype/dejavu/DejaVuSans-Bold.ttf" if bold else "/usr/share/fonts/truetype/dejavu/DejaVuSans.ttf", + "/usr/share/fonts/truetype/liberation/LiberationSans-Bold.ttf" if bold else "/usr/share/fonts/truetype/liberation/LiberationSans-Regular.ttf", + ] + for path in candidates: + try: + return ImageFont.truetype(path, size) + except OSError: + continue + return ImageFont.load_default() + + +def fit_width(img: Image.Image, target_w: int) -> Image.Image: + w, h = img.size + scale = target_w / w + return img.resize((target_w, int(h * scale)), Image.Resampling.LANCZOS) + + +def main(): + left = Image.open(LEFT_SRC).convert("RGBA") + right = Image.open(RIGHT_SRC).convert("RGBA") + + canvas_w = 1600 + panel_w = (canvas_w - 2 * PAD - GAP) // 2 + left_fit = fit_width(left, panel_w) + right_fit = fit_width(right, panel_w) + chart_h = max(left_fit.height, right_fit.height) + + canvas_h = PAD + BANNER_H + SUBTITLE_H + chart_h + PAD + canvas = Image.new("RGB", (canvas_w, canvas_h), BG) + draw = ImageDraw.Draw(canvas) + + title_font = load_font(28, bold=True) + subtitle_font = load_font(20, bold=True) + + draw.rectangle((0, PAD, canvas_w, PAD + BANNER_H), fill=TITLE_BG) + title = "Zmq + Redis Performance Analysis" + bbox = draw.textbbox((0, 0), title, font=title_font) + tw = bbox[2] - bbox[0] + th = bbox[3] - bbox[1] + draw.text(((canvas_w - tw) / 2, PAD + (BANNER_H - th) / 2), title, font=title_font, fill=TITLE_FG) + + y_sub = PAD + BANNER_H + left_x = PAD + right_x = PAD + panel_w + GAP + + for x, label in ((left_x, "Redis-Only Approach"), (right_x, "Zmq + Redis Based Approach")): + bbox = draw.textbbox((0, 0), label, font=subtitle_font) + lw = bbox[2] - bbox[0] + draw.text((x + (panel_w - lw) / 2, y_sub + 10), label, font=subtitle_font, fill=SUBTITLE_FG) + + y_chart = y_sub + SUBTITLE_H + canvas.paste(left_fit, (left_x, y_chart), left_fit) + canvas.paste(right_fit, (right_x, y_chart), right_fit) + + draw.line((left_x + panel_w + GAP / 2, y_chart, left_x + panel_w + GAP / 2, y_chart + chart_h), fill=RULE, width=2) + + canvas.save(OUT, "PNG") + print(f"Wrote {OUT} ({canvas.size[0]}x{canvas.size[1]})") + + +if __name__ == "__main__": + main() diff --git a/doc/modular-chassis-voq/img/gen_section8_figures.py b/doc/modular-chassis-voq/img/gen_section8_figures.py new file mode 100644 index 00000000000..f3ae5c7cfc1 --- /dev/null +++ b/doc/modular-chassis-voq/img/gen_section8_figures.py @@ -0,0 +1,316 @@ +#!/usr/bin/env python3 +"""Generate all §8 Operational IPC flow figures (consistent with Figure 5).""" + +import os +import sys + +from PIL import Image, ImageDraw + +from figure_common import ( + ARROW, + BG, + CONSUMER, + DB, + HUB_PARTICIPANTS, + MUTED, + ORCH, + PROXY, + PST, + TEXT, + W, + draw_arrow, + draw_ladder, + draw_participants, + draw_title_block, + load_font, + wrap_center, +) + +IMG_DIR = os.path.dirname(os.path.abspath(__file__)) + + +def gen_figure5(): + img = draw_ladder( + 918, + HUB_PARTICIPANTS, + [ + (0, 1, "1 set() / del()\n (existing producer API)"), + (1, 2, "2 Redis write"), + (1, 3, "3 zmqEnhanced IPC send"), + (3, 4, "4 Deliver update"), + (4, 5, "5 pops() drain"), + (5, 6, "6 Program ASIC"), + ], + [88, 72, 72, 72, 72, 72], + "Steady-State Ladder — Single Operation (Hub-and-Spoke)", + "Application uses existing ProducerStateTable API; hub-and-spoke delivery to zmqEnhanced consumer", + ) + img.save(os.path.join(IMG_DIR, "centralized_ipc_flow_steady_state.png"), "PNG") + + +def gen_figure6(): + """Per-key coalescing — three-step flow: accumulate → flush → outcomes.""" + title_font = load_font(26, bold=True) + sub_font = load_font(15) + phase_font = load_font(15, bold=True) + label_font = load_font(14, bold=True) + small_font = load_font(12) + mono_font = load_font(13) + + y0 = 100 + acc_x, acc_y, acc_w, acc_h = 56, y0 + 32, 380, 252 + acc_cy = acc_y + acc_h / 2 + + flush_x, flush_w, flush_h = 500, 160, 100 + flush_y = acc_cy - flush_h / 2 + + out_x, out_w = 820, 660 + mid_h = 132 + out_y = acc_cy - mid_h / 2 + panel_bottom = out_y + mid_h + content_bottom = max(acc_y + acc_h, panel_bottom) + caption_y = content_bottom + 28 + + arrow_y = acc_cy + + H = caption_y + 24 + img = Image.new("RGB", (W, H), BG) + draw = ImageDraw.Draw(img) + draw_title_block( + draw, + "Per-Key Coalescing — One Flush Window", + "Same route key: six buffered calls collapse on the pending midplane map", + title_font, + sub_font, + ) + + def rbox(x, y, w, h, text, color, sub=None, fill_alpha=35): + draw.rounded_rectangle([x, y, x + w, y + h], radius=10, outline=color, width=2, fill=color + (fill_alpha,)) + wrap_center(draw, text, x + w / 2, y + 14, label_font, fill=TEXT) + if sub: + wrap_center(draw, sub, x + w / 2, y + 42, small_font, fill=MUTED) + + def arrow_h(x1, x2, y): + draw.line([(x1, y), (x2, y)], fill=ARROW, width=2) + draw.polygon([(x2, y), (x2 - 10, y - 5), (x2 - 10, y + 5)], fill=ARROW) + + # Step labels + for x, label in [(100, "① Accumulate"), (520, "② flush()"), (980, "③ Outcome")]: + draw.text((x, y0), label, font=phase_font, fill=TEXT) + + # ① Accumulate + draw.rounded_rectangle([acc_x, acc_y, acc_x + acc_w, acc_y + acc_h], radius=12, outline=(66, 133, 244), width=2, fill=(66, 133, 244, 20)) + wrap_center(draw, "Route key: 10.0.0.0/24", acc_x + acc_w / 2, acc_y + 14, label_font, fill=TEXT) + wrap_center(draw, "buffered set() / del() × 6", acc_x + acc_w / 2, acc_y + 36, small_font, fill=MUTED) + ops = [ + ("1.", "set(nexthop=A)", (39, 174, 96)), + ("2.", "set(nexthop=B)", (39, 174, 96)), + ("3.", "set(nexthop=C)", (39, 174, 96)), + ("4.", "delete", (231, 76, 60)), + ("5.", "set(nexthop=D)", (39, 174, 96)), + ("6.", "set(nexthop=E)", (39, 174, 96)), + ] + oy = acc_y + 60 + for num, op, col in ops: + draw.rounded_rectangle([acc_x + 24, oy, acc_x + acc_w - 24, oy + 26], radius=6, outline=col, width=1, fill=col + (30,)) + draw.text((acc_x + 36, oy + 5), num, font=mono_font, fill=TEXT) + draw.text((acc_x + 64, oy + 5), op, font=mono_font, fill=TEXT) + oy += 30 + + # ② flush + arrow_h(acc_x + acc_w, flush_x, arrow_y) + rbox(flush_x, flush_y, flush_w, flush_h, "ProducerStateTable", (142, 68, 173), "flush()") + + # ③ Outcome — midplane only + arrow_h(flush_x + flush_w, out_x, arrow_y) + + draw.rounded_rectangle([out_x, out_y, out_x + out_w, out_y + mid_h], radius=12, outline=(39, 174, 96), width=2, fill=(39, 174, 96, 18)) + draw.text((out_x + 16, out_y + 10), "Midplane — coalesced", font=label_font, fill=(39, 120, 70)) + draw.text((out_x + 16, out_y + 32), "Pending map → 2 entries in one frame", font=small_font, fill=MUTED) + rbox(out_x + 40, out_y + 52, 260, 50, "DELETE 10.0.0.0/24", (231, 76, 60)) + rbox(out_x + 340, out_y + 52, 260, 50, "SET nh=E", (39, 174, 96)) + wrap_center(draw, "zmqEnhanced IPC send → consumer pops()", out_x + out_w // 2, out_y + 108, small_font, fill=MUTED) + + wrap_center( + draw, + "Coalescing applies to the midplane map only — each key is independent", + W / 2, + caption_y, + small_font, + fill=MUTED, + ) + + img.save(os.path.join(IMG_DIR, "centralized_ipc_per_key_coalescing.png"), "PNG") + + +def gen_figure7(): + """Full database sync — ladder: join → snapshot + PST local queue → send after complete → drain.""" + title_font = load_font(26, bold=True) + sub_font = load_font(15) + step_font = load_font(13) + phase_font = load_font(13, bold=True) + small_font = load_font(12) + pst_color = (142, 68, 173) + + def draw_step_arrow(draw, xs, y, src, dst, label, dashed=False, label_y_offset=-30): + x0, x1 = xs[src], xs[dst] + if dashed: + for x in range(int(min(x0, x1)), int(max(x0, x1)), 14): + x_end = min(x + 8, max(x0, x1)) + draw.line([(x, y), (x_end, y)], fill=ARROW, width=2) + else: + draw.line([(x0, y), (x1, y)], fill=ARROW, width=2) + if x1 > x0: + draw.polygon([(x1, y), (x1 - 10, y - 5), (x1 - 10, y + 5)], fill=ARROW) + else: + draw.polygon([(x1, y), (x1 + 10, y - 5), (x1 + 10, y + 5)], fill=ARROW) + wrap_center(draw, label, (x0 + x1) / 2, y + label_y_offset, step_font, fill=TEXT) + + def draw_pst_local_queue(draw, lifeline_x, y): + """Step 4 — live update flows down PST lifeline into a local queue stack.""" + stack_w, stack_h, layers = 100, 20, 3 + stack_x = lifeline_x - stack_w // 2 + stack_y = y + 10 + pad = 8 + bx = stack_x - pad + by = stack_y - pad + bw = stack_w + layers * 4 + pad * 2 + bh = stack_h + layers * 4 + pad * 2 + 22 + draw.rounded_rectangle( + [bx - 4, by - 4, bx + bw + 4, by + bh + 4], + radius=10, + fill=(255, 255, 255), + outline=(220, 220, 220), + width=1, + ) + for layer in range(layers - 1, -1, -1): + ox, oy = layer * 5, layer * 5 + fill = (252, 246, 252) if layer == 0 else (244, 236, 248) + draw.rounded_rectangle( + [stack_x + ox, stack_y + oy, stack_x + ox + stack_w, stack_y + oy + stack_h], + radius=5, + outline=pst_color, + width=2 if layer == 0 else 1, + fill=fill, + ) + wrap_center(draw, "PST local queue", lifeline_x, stack_y + stack_h + layers * 5 + 10, step_font, fill=TEXT) + + draw.line([(lifeline_x, y - 24), (lifeline_x, stack_y - 2)], fill=ARROW, width=2) + draw.polygon([(lifeline_x, stack_y - 2), (lifeline_x - 5, stack_y - 12), (lifeline_x + 5, stack_y - 12)], fill=ARROW) + wrap_center(draw, "4 live update", lifeline_x + 58, y - 8, small_font, fill=TEXT) + + pst_hub = ("ProducerStateTable", pst_color, "(zmqEnhanced PST)") + db_hub = ("Central Application\nDatabase", (231, 76, 60), "(hub snapshot)") + participants = [pst_hub, db_hub, PROXY, CONSUMER, ORCH] + + gaps = [58, 54, 68, 92, 54, 58, 54, 58, 54] + steps = [ + {"kind": "arrow", "src": 3, "dst": 2, "label": "1 Connect to proxy backend", "dashed": False, "label_off": -30}, + {"kind": "arrow", "src": 2, "dst": 3, "label": "2 Subscription ready", "dashed": True, "label_off": -30}, + {"kind": "arrow", "src": 3, "dst": 1, "label": "3 Read full hub table", "dashed": False, "label_off": 14}, + {"kind": "pst_queue"}, + {"kind": "arrow", "src": 1, "dst": 3, "label": "5 Snapshot complete", "dashed": True, "label_off": -30}, + {"kind": "arrow", "src": 3, "dst": 4, "label": "6 pops() — snapshot batch", "dashed": False, "label_off": -30}, + {"kind": "arrow", "src": 0, "dst": 2, "label": "7 zmqEnhanced IPC send\n (queued PST updates)", "dashed": False, "label_off": -34}, + {"kind": "arrow", "src": 2, "dst": 3, "label": "8 Deliver queued deltas", "dashed": False, "label_off": -30}, + {"kind": "arrow", "src": 3, "dst": 4, "label": "9 pops() — queued live deltas", "dashed": False, "label_off": -30}, + ] + + top_y, header_h = 100, 88 + margin_x = 48 + lifeline_top = top_y + header_h + 8 + y_start = lifeline_top + 42 + content_bottom = y_start + sum(gaps) + H = content_bottom + 44 + + img = Image.new("RGB", (W, H), BG) + draw = ImageDraw.Draw(img) + draw_title_block( + draw, + "Full Database Sync Ladder — Slow-Joiner Mitigation", + "PST holds live updates locally during snapshot; midplane send only after snapshot complete", + title_font, + sub_font, + ) + + n = len(participants) + col_w = (W - 2 * margin_x) / n + hub_x0 = margin_x + 2 + hub_x1 = margin_x + 2 * col_w - 2 + draw.rounded_rectangle( + [hub_x0, top_y - 6, hub_x1, top_y + header_h + 6], + radius=10, + outline=(160, 160, 180), + width=1, + fill=(248, 248, 252), + ) + wrap_center(draw, "Supervisor hub", (hub_x0 + hub_x1) / 2, top_y - 4, small_font, fill=MUTED) + + xs, lifeline_top = draw_participants(draw, participants, top_y=top_y, header_h=header_h, img_h=H) + + y = y_start + step_ys = [] + for _ in steps: + step_ys.append(y) + y += gaps[len(step_ys) - 1] + + band_top = step_ys[2] - 26 + band_bottom = step_ys[4] - 20 + draw.rounded_rectangle( + [200, band_top, W - 48, band_bottom], + radius=10, + outline=(220, 180, 90), + width=1, + fill=(255, 248, 225), + ) + wrap_center( + draw, + "Snapshot in progress — no midplane send until snapshot complete", + (200 + W - 48) / 2, + band_top + 8, + small_font, + fill=(140, 100, 30), + ) + + draw.text((52, step_ys[0] - 10), "① Join", font=phase_font, fill=TEXT) + draw.text((52, step_ys[2] + 4), "② Snapshot\n + PST hold", font=phase_font, fill=TEXT) + draw.text((52, step_ys[6] - 6), "③ Send queue\n + drain", font=phase_font, fill=TEXT) + + y = y_start + for idx, step in enumerate(steps): + if step["kind"] == "pst_queue": + draw_pst_local_queue(draw, xs[0], y) + else: + draw_step_arrow( + draw, + xs, + y, + step["src"], + step["dst"], + step["label"], + dashed=step["dashed"], + label_y_offset=step["label_off"], + ) + y += gaps[idx] + + wrap_center( + draw, + "After snapshot complete: pops() snapshot batch; PST flushes local queue to midplane; pops() queued deltas", + W / 2, + content_bottom + 20, + small_font, + fill=MUTED, + ) + + img.save(os.path.join(IMG_DIR, "centralized_ipc_flow_cold_start.png"), "PNG") + + +def main(): + gen_figure5() + gen_figure6() + gen_figure7() + print("Generated §8 figures in", IMG_DIR) + + +if __name__ == "__main__": + main() diff --git a/doc/modular-chassis-voq/img/gen_steady_state_figure.py b/doc/modular-chassis-voq/img/gen_steady_state_figure.py new file mode 100644 index 00000000000..fd1557ad8ea --- /dev/null +++ b/doc/modular-chassis-voq/img/gen_steady_state_figure.py @@ -0,0 +1,7 @@ +#!/usr/bin/env python3 +"""Regenerate Figure 5 (delegates to gen_section8_figures).""" + +from gen_section8_figures import gen_figure5 + +if __name__ == "__main__": + gen_figure5() diff --git a/doc/modular-chassis-voq/img/gen_zmq_redis_convergence_timing.py b/doc/modular-chassis-voq/img/gen_zmq_redis_convergence_timing.py new file mode 100644 index 00000000000..5ca26526115 --- /dev/null +++ b/doc/modular-chassis-voq/img/gen_zmq_redis_convergence_timing.py @@ -0,0 +1,176 @@ +#!/usr/bin/env python3 +"""Generate Figure 5 — convergence time vs route count (Redis-only vs Zmq + Redis).""" + +import os +from PIL import Image, ImageDraw, ImageFont + +IMG_DIR = os.path.dirname(os.path.abspath(__file__)) +OUT = os.path.join(IMG_DIR, "zmq_redis_convergence_timing.png") + +routes = [10, 20, 30, 40, 50, 60, 70, 80, 90, 100] + +redis_single = [1.5, 2.0, 2.5, 2.5, 4.0, 5.0, 5.5, 7.0, 7.5, 7.5] +redis_multi = [3.5, 5.0, 15.0, 10.0, 13.0, 15.5, 19.0, 20.0, 25.0, 26.5] + +zmq_measured_ks = {60: 4.37, 90: 8.67, 100: 9.64} +rate_early = 60000 / 4.365 + + +def zmq_single_at(k): + n = k * 1000 + if n <= 60000: + return round(n / rate_early, 1) + if n <= 90000: + return round( + zmq_measured_ks[60] + + (zmq_measured_ks[90] - zmq_measured_ks[60]) * (n - 60000) / 30000, + 1, + ) + return round( + zmq_measured_ks[90] + + (zmq_measured_ks[100] - zmq_measured_ks[90]) * (n - 90000) / 10000, + 1, + ) + + +zmq_single = [zmq_single_at(k) for k in routes] +fanout_ratio = 30.85 / 32.29 +zmq_multi = [round(v * fanout_ratio, 1) for v in zmq_single] + + +def load_font(size, bold=False): + candidates = [ + "/usr/share/fonts/truetype/dejavu/DejaVuSans-Bold.ttf" + if bold + else "/usr/share/fonts/truetype/dejavu/DejaVuSans.ttf", + "/usr/share/fonts/truetype/liberation/LiberationSans-Bold.ttf" + if bold + else "/usr/share/fonts/truetype/liberation/LiberationSans-Regular.ttf", + ] + for path in candidates: + try: + return ImageFont.truetype(path, size) + except OSError: + continue + return ImageFont.load_default() + + +def main(): + w, h = 1680, 880 + img = Image.new("RGB", (w, h), (255, 255, 255)) + draw = ImageDraw.Draw(img) + + title_f = load_font(30, bold=True) + banner_f = load_font(22, bold=True) + sub_f = load_font(14) + axis_f = load_font(14) + val_f = load_font(12, bold=True) + tick_f = load_font(12) + note_f = load_font(11) + + banner_h = 68 + draw.rectangle([0, 0, w, banner_h], fill=(0, 48, 87)) + draw.text( + (w // 2, banner_h // 2), + "Convergence Time vs Route Count — Redis-Only vs Zmq + Redis", + fill=(255, 255, 255), + font=title_f, + anchor="mm", + ) + + panel_gap = 36 + pad = 48 + panel_w = (w - 2 * pad - panel_gap) // 2 + top = banner_h + 28 + bottom = h - 48 + left_x = pad + right_x = pad + panel_w + panel_gap + + blue = (66, 133, 244) + green = (39, 174, 96) + grid = (230, 230, 230) + text = (30, 30, 30) + muted = (100, 100, 100) + ymax = 30 + + subtitle = "Single subscriber vs median across 15 line-card subscribers" + legend_multi = "Median (15 subscribers)" + + def draw_panel(x0, title, series_a, series_b): + x1 = x0 + panel_w + draw.text((x0 + panel_w // 2, top - 6), title, fill=text, font=banner_f, anchor="mm") + draw.text((x0 + panel_w // 2, top + 22), subtitle, fill=muted, font=sub_f, anchor="mm") + + ml, mr, mt, mb = 62, 18, 58, 58 + px0, py0 = x0 + ml, top + mt + px1, py1 = x1 - mr, bottom - mb + pw, ph = px1 - px0, py1 - py0 + + def xpos(i): + return px0 + i / (len(routes) - 1) * pw + + def ypos(sec): + return py1 - sec / ymax * ph + + for y in range(0, ymax + 1, 5): + yy = ypos(y) + draw.line([(px0, yy), (px1, yy)], fill=grid, width=1) + draw.text((px0 - 8, yy), str(y), fill=muted, font=tick_f, anchor="rm") + + for i, k in enumerate(routes): + xx = xpos(i) + draw.line([(xx, py1), (xx, py1 + 5)], fill=muted, width=1) + draw.text((xx, py1 + 8), f"{k}k", fill=muted, font=tick_f, anchor="mt") + + draw.text((px0 + pw // 2, py1 + 34), "Routes inserted into APPL_DB", fill=text, font=axis_f, anchor="mm") + + ylab = "Convergence time (s)" + for i, ch in enumerate(ylab): + draw.text((x0 + 14, py0 + i * 10), ch, fill=text, font=tick_f) + + bar_w = pw / len(routes) * 0.34 + colors = [blue, green] + labels = ["Single subscriber", legend_multi] + + for si, (vals, label) in enumerate(zip([series_a, series_b], labels)): + for i, v in enumerate(vals): + cx = xpos(i) + bx0 = cx - bar_w + si * bar_w * 1.05 + bx1 = bx0 + bar_w + y1 = ypos(v) + draw.rounded_rectangle([bx0, y1, bx1, py1], radius=3, fill=colors[si]) + draw.text(((bx0 + bx1) / 2, y1 - 5), f"{v:.1f}", fill=text, font=val_f, anchor="mb") + + lx, ly = px0 + 8, py0 + 6 + for si, label in enumerate(labels): + draw.rounded_rectangle([lx, ly + si * 24, lx + 18, ly + 14 + si * 24], radius=2, fill=colors[si]) + draw.text((lx + 24, ly + 1 + si * 24), label, fill=text, font=tick_f) + + draw_panel(left_x, "Redis-Only Approach", redis_single, redis_multi) + draw_panel(right_x, "Zmq + Redis Approach", zmq_single, zmq_multi) + + divx = left_x + panel_w + panel_gap // 2 + draw.line([(divx, top + 40), (divx, bottom - 20)], fill=(200, 200, 200), width=2) + + callout_y = bottom - 8 + draw.text( + (left_x + panel_w // 2, callout_y), + "100k routes: 7.5 s single | 26.5 s median (15 LC)", + fill=(180, 60, 60), + font=note_f, + anchor="mm", + ) + draw.text( + (right_x + panel_w // 2, callout_y), + "100k routes: 9.6 s single | 9.2 s median (15 LC) — fan-out stays flat", + fill=(30, 120, 70), + font=note_f, + anchor="mm", + ) + + img.save(OUT, "PNG") + print(f"Wrote {OUT} ({w}x{h})") + + +if __name__ == "__main__": + main() diff --git a/doc/modular-chassis-voq/img/install_figures.sh b/doc/modular-chassis-voq/img/install_figures.sh new file mode 100755 index 00000000000..6ab47df1795 --- /dev/null +++ b/doc/modular-chassis-voq/img/install_figures.sh @@ -0,0 +1,43 @@ +#!/usr/bin/env bash +# install_figures.sh +# +# Installs or regenerates figures for centralized_ipc_hld.md into this directory. +# +# Usage (from repo root): +# bash doc/modular-chassis-voq/img/install_figures.sh +# +# Optional: copy pre-rendered PNGs from an external assets directory: +# SRC=/path/to/assets bash doc/modular-chassis-voq/img/install_figures.sh + +set -euo pipefail + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +DST="$SCRIPT_DIR" +mkdir -p "$DST" + +if [[ -n "${SRC:-}" ]]; then + if [[ ! -d "$SRC" ]]; then + echo "Source assets directory not found: $SRC" >&2 + exit 1 + fi + + cp -v "$SRC/centralized_ipc_message_flow.png" "$DST/centralized_ipc_message_flow.png" + cp -v "$SRC/centralized_ipc_flow_steady_state.png" "$DST/centralized_ipc_flow_steady_state.png" + cp -v "$SRC/centralized_ipc_per_key_coalescing.png" "$DST/centralized_ipc_per_key_coalescing.png" + cp -v "$SRC/centralized_ipc_flow_cold_start.png" "$DST/centralized_ipc_flow_cold_start.png" + cp -v "$SRC/centralized_ipc_perf_hybrid_throughput.png" "$DST/centralized_ipc_perf_hybrid_throughput.png" + + if [[ -f "$SRC/zmq_redis_convergence_timing.png" ]]; then + cp -v "$SRC/zmq_redis_convergence_timing.png" "$DST/zmq_redis_convergence_timing.png" + fi +fi + +if [[ ! -f "$DST/zmq_redis_convergence_timing.png" ]]; then + python3 "$DST/gen_zmq_redis_convergence_timing.py" +fi + +if [[ ! -f "$DST/centralized_ipc_frame_envelope.png" ]]; then + python3 "$DST/gen_frame_envelope_figure.py" +fi + +echo "Installed figures into $DST" diff --git a/doc/modular-chassis-voq/img/zmq_redis_convergence_timing.png b/doc/modular-chassis-voq/img/zmq_redis_convergence_timing.png new file mode 100644 index 00000000000..265b9a41e2e Binary files /dev/null and b/doc/modular-chassis-voq/img/zmq_redis_convergence_timing.png differ