Hosting Integration and Health Model

docs/sonic-agentd-container.md

@@ -20,10 +20,12 @@ is the recommended mode for this agent.
 ```bash
 docker run --rm --network host \
     -v /var/run/redis/redis.sock:/var/run/redis/redis.sock \
+    -v /run/tacacs:/run/tacacs \
     -p 127.0.0.1:49:49 \
     ghcr.io/authscaffold/tacacsrs-agentd:2026.706.1 \
     --sonic \
     --service-mode both \
+    --host-integration none \
     --proxy-endpoint 127.0.0.1:49 \
     -vv
 ```
@@ -37,7 +39,9 @@ The flags mean:
 | `-p 127.0.0.1:49:49` | Harmless with host networking, but ignored by Docker because there is no separate container network namespace to publish from. |
 | `--sonic` | Load TACACS+ server configuration from SONiC CONFIG_DB, database `4`. |
 | `--service-mode both` | Run both the local client API and the raw TACACS+ proxy service. |
+| `--host-integration none` | Disable systemd notification inside the container. |
 | `--proxy-endpoint 127.0.0.1:49` | Bind the proxy on loopback only, so local SONiC clients can connect without exposing port `49` on external interfaces. |
+| `-v /run/tacacs:/run/tacacs` | Expose the Client API UDS and standard gRPC health service to host consumers and exec probes. |
 | `-vv` | Enable info-level logging. |
 
 Because `-p` is ignored in host networking mode, the command can also be written
@@ -46,9 +50,11 @@ without the publish flag:
 ```bash
 docker run --rm --network host \
     -v /var/run/redis/redis.sock:/var/run/redis/redis.sock \
+    -v /run/tacacs:/run/tacacs \
     ghcr.io/authscaffold/tacacsrs-agentd:2026.706.1 \
     --sonic \
     --service-mode both \
+    --host-integration none \
     --proxy-endpoint 127.0.0.1:49 \
     -vv
 ```
@@ -114,14 +120,22 @@ At least one `TACPLUS_SERVER` row must be present for upstream TACACS+ traffic.
 For the schema mapping and Redis notification details, see
 [SONiC ConfigDB Integration](sonic-configdb-integration.md).
 
+The process and Client API listener start even if Redis is temporarily unavailable. Startup and readiness remain not serving while the daemon retries ConfigDB with capped jittered backoff. A valid snapshot makes the same process ready without a restart. Upstream reachability does not gate readiness.
+
+Use the standard exec probe in the image:
+
+```bash
+tacacsrs-agent-health \
+    --endpoint /run/tacacs/tacacs.sock \
+    --check readiness \
+    --timeout-seconds 2
+```
+
 The startup log should include lines similar to:
 
 ```text
 Configured SONiC ConfigDB datastore: url='unix:///var/run/redis/redis.sock?db=4', db=4
-Initial configuration loaded from datastore 'sonic-configdb'
-Upstream servers: 1 configured, probe interval: 30s
 TACACS+ proxy endpoint: Tcp(127.0.0.1:49)
 ```
 
-If CONFIG_DB changes should be picked up live, enable Redis keyspace
-notifications as described in the ConfigDB integration guide.
+Enable Redis keyspace notifications as described in the ConfigDB integration guide. If the subscription fails or ends, the daemon retains its known-good configuration, reloads to cover missed events, and resubscribes.

docs/sonic-configdb-integration.md

@@ -252,25 +252,22 @@ cargo run -p tacacsrs-agentd -- \
 ```
 
 Mutating the Redis rows should produce the documented configuration-change log
-message. The current daemon observes changes but does not hot-swap upstream
-connections in place; restart `tacacsrs-agentd` to apply a changed server set.
+message. The daemon atomically applies each valid filtered snapshot to new
+sessions without restarting. In-flight sessions keep their existing server-set
+snapshot and connection handles.
 
 ## Hot reload behaviour
 
-When a TACPLUS-prefixed key changes in CONFIG_DB, the bridge:
+When a TACPLUS-prefixed key changes in CONFIG_DB, the runtime:
 
 1. Coalesces additional changes that arrive within a short debounce window.
 2. Re-reads the full TACPLUS / TACPLUS_SERVER tables.
 3. Validates the new snapshot against the YANG schema.
-4. Emits a `ConfigChange` event carrying the new snapshot and a delta
-   describing which servers were added, removed, or modified.
-
-The current daemon does not yet hot-swap upstream connections in place: it
-records the change and asks the operator to restart the service to apply
-the new configuration. This is the smallest safe step that keeps in-flight
-TACACS+ sessions intact, and the plumbing is shaped so that a future
-implementation can replace the listener body with an atomic state-rebuild
-without changing the surrounding lifecycle or the datastore contract.
+4. Emits a typed changed or rejected event.
+5. Filters proxy self-loops and validates the complete candidate.
+6. Atomically replaces the server set for new sessions while preserving unchanged cached connections.
+
+Invalid candidates leave the previous known-good configuration active and mark runtime health stale/degraded. If Redis is unavailable at process startup, enabled listeners still bind and liveness serves while startup/readiness remain not serving. The daemon retries with capped jittered backoff. Subscription setup failures and ended streams mark the snapshot stale, trigger a fresh load to cover missed changes, and resubscribe without exiting.
 
 ## Operational commands
 

docs/tacacs-plus-tls-transition.md

@@ -53,13 +53,12 @@ Keep these in the legacy client configuration:
 
 ## Shared Secret Compatibility
 
-Proxy mode currently uses the selected upstream server's TACACS+ shared secret
-for the downstream client connection too. That means:
+The downstream client hop has its own TACACS+ obfuscation policy and does not follow the selected upstream server:
 
-- Keep the legacy client `secret=` value and the `tacacsrs-agentd --shared-secret` value aligned.
-- Use the same TACACS+ shared secret across the upstream servers used by one proxy deployment.
-- Avoid a cutover where the local client secret differs from the upstream server secret unless the proxy gains a separate downstream-secret setting.
-- If the upstream TLS server disables TACACS+ packet obfuscation entirely, verify whether the legacy client can send unencrypted TACACS+ packets before relying on proxy mode.
+- Outside SONiC mode, keep the legacy client `secret=` value aligned with `tacacsrs-agentd --proxy-shared-secret`.
+- In SONiC mode, configure the local loopback proxy row with the legacy client secret. The daemon filters that self-targeting row from the upstream set and uses its resolved `passkey` for downstream obfuscation.
+- If neither source provides a downstream secret, verify that the legacy client can send TACACS+ packets with the unencrypted flag.
+- Upstream servers may use different classic shared secrets or TLS credentials; failover does not change the local client secret.
 
 TLS protects the upstream connection, but it does not replace the downstream
 TACACS+ packet format expected by `pam_tacplus` or `audisp-tacplus`.

docs/tacacsrs-agentd.md

@@ -61,6 +61,7 @@ tacon --service-endpoint /run/tacacs/tacacs.sock \
 | `--proxy-endpoint <ENDPOINT>` | *(disabled)* | Optional TACACS+ proxy listener on a Unix socket path or loopback TCP address |
 | `--service-mode <MODE>` | `client-api`, or `both` when `--proxy-endpoint` is set | Runtime services to host: `client-api`, `tacacs-proxy`, or `both` |
 | `--socket-mode <MODE>` | `660` | File permission mode for the Unix socket (octal) |
+| `--host-integration <MODE>` | `auto` | Host adapter: `auto`, `none`, or strict `systemd` |
 
 ### Runtime Service Modes
 
@@ -92,7 +93,7 @@ Proxy mode is deliberately packet-transparent:
 
 Proxy TCP endpoints must be loopback addresses. Unix domain socket endpoints use the same `--socket-mode` value as the IPC listener. When `client-api` and `tacacs-proxy` run together, the proxy endpoint must be different from `--listen-endpoint`.
 
-The downstream TACACS+ shared-secret behavior follows the upstream server selected for that connection. If the selected upstream server has a `shared-secret`, the proxy uses that secret to deobfuscate downstream packets and obfuscate replies. If the selected upstream server has no shared secret, downstream packets must be sent with the unencrypted flag. When configured upstream servers use different shared secrets, a reconnect or failover can select a server with a different downstream secret; keep upstream shared secrets identical when using proxy mode.
+Downstream TACACS+ obfuscation is independent of the selected upstream transport. In SONiC mode, the daemon filters rows that target its own loopback proxy and uses the highest-priority matching row's resolved `passkey` for the local hop. Outside SONiC mode, use `--proxy-shared-secret`. If neither source provides a local-hop secret, downstream clients must send unobfuscated TACACS+ packets. Upstream failover never changes the downstream secret.
 
 ### Upstream Encryption
 
@@ -173,6 +174,44 @@ When multiple IPC requests arrive simultaneously during a reconnect, only one co
 
 On startup the daemon attempts to connect to servers in order and stops at the first success. This prevents connection storms when many instances start simultaneously (e.g. during a fleet rollout). If no server is reachable at startup, the daemon still starts and requests will retry on demand.
 
+SONiC ConfigDB is supervised differently from local CLI or file input. The daemon binds enabled listeners with an empty runtime configuration, reports startup/readiness as not serving, and retries ConfigDB with capped jittered backoff. When Redis becomes available, the same process applies the first valid snapshot and becomes ready. Subscription failures and ended streams trigger a fresh load before resubscription; invalid candidates leave the previous known-good runtime configuration active and mark health degraded.
+
+## Health and Probes
+
+The Client API endpoint also serves the standard `grpc.health.v1.Health` protocol. No custom health protobuf or additional network listener is used.
+
+| Service name | Meaning |
+|---|---|
+| `tacacsrs.agent.health.v1.Startup` | A validated snapshot is applied and every enabled listener is bound |
+| `tacacsrs.agent.health.v1.Liveness` | The process is starting or serving and the Client API can answer |
+| `tacacsrs.agent.health.v1.Readiness` | Startup is complete and at least one eligible upstream is configured |
+| *(empty service name)* | Same as readiness |
+| `tacacsrs.agent.v1.TacacsAgent` | Same as readiness for the business RPC service |
+
+Current upstream reachability is diagnostic and does not gate readiness or liveness. During shutdown all names become `NOT_SERVING` before listeners stop accepting work.
+
+Use the packaged probe for Kubernetes exec probes and local diagnostics:
+
+```bash
+tacacsrs-agent-health \
+    --endpoint /run/tacacs/tacacs.sock \
+    --check readiness \
+    --timeout-seconds 2
+```
+
+| Exit code | Meaning |
+|---|---|
+| `0` | The selected health name is `SERVING` |
+| `1` | The selected health name is not serving or unknown |
+| `2` | Probe invocation or endpoint syntax is invalid |
+| `3` | Endpoint, timeout, transport, or protocol failure |
+
+The health endpoint exists only when `client-api` is enabled. A proxy-only process still publishes runtime state to logs and systemd but cannot use this gRPC probe.
+
+## Host Integration
+
+`--host-integration auto` selects systemd only when `NOTIFY_SOCKET` is present. `none` never invokes a host API and is the container setting. Explicit `systemd` requires both `NOTIFY_SOCKET` and `systemd-notify`; missing prerequisites and runtime notification failures are fatal. Systemd receives a waiting status before readiness, `READY=1` exactly once, sanitized degraded status updates, and `STOPPING=1` before listener drain.
+
 ## IPC Protocol
 
 The daemon communicates with clients via gRPC over Unix domain sockets (Linux) or loopback TCP (other platforms). The protocol is defined in protobuf:
@@ -199,12 +238,14 @@ After=network-online.target
 Wants=network-online.target
 
 [Service]
-Type=simple
+Type=notify
+NotifyAccess=all
 ExecStart=/usr/local/bin/tacacsrs-agentd \
     --server-addr tacacs1.example.com:49 \
     --server-addr tacacs2.example.com:49 \
     --listen-endpoint /run/tacacs/tacacs.sock \
     --socket-mode 660 \
+    --host-integration systemd \
     --shared-secret "shared_secret" \
     --preferred-probe-interval-seconds 30
 Restart=on-failure

executables/tacacsrs_agentd/Cargo.toml

@@ -8,11 +8,13 @@ description = "Central TACACS+ client service for local consumers"
 
 [dependencies]
 anyhow = { version = "1.0.102", features = ["backtrace"] }
+async-trait = "0.1.89"
 clap = { version = "4.6.1", features = ["derive"] }
 console-subscriber = { version = "0.5.0", optional = true }
 env_logger = "0.11.10"
 futures-util = { version = "0.3", default-features = false }
 log = "0.4"
+rand = "0.10.0"
 rustls-pki-types = "1.14.1"
 tacacsrs-agent-client = { path = "../../libraries/tacacsrs_agent_client" }
 tacacsrs-agent = { path = "../../libraries/tacacsrs_agent" }
@@ -22,11 +24,17 @@ tacacsrs-datastore = { path = "../../libraries/tacacsrs_datastore" }
 tacacsrs-networking = { path = "../../libraries/tacacsrs_networking" }
 tacacsrs-sonic = { path = "../../libraries/tacacsrs_sonic" }
 tokio = { version = "1.52.3", features = ["full"] }
+tokio-stream = "0.1.18"
+tokio-util = "0.7.18"
+tonic-health = "0.14.6"
 
 [features]
 default = []
 console = ["console-subscriber"]
 
+[dev-dependencies]
+tonic = "0.14.6"
+
 [build-dependencies]
 clap = { version = "4.6.1", features = ["derive"] }
 clap_mangen = "0.3.0"
@@ -54,6 +62,7 @@ priority = "optional"
 maintainer-scripts = "debian/maintainer-scripts"
 assets = [
     ["target/release/tacacsrs-agentd", "usr/sbin/", "755"],
+    ["target/release/tacacsrs-agent-health", "usr/bin/", "755"],
     ["debian/tacacsrs-agentd-config.ini", "etc/tacacsrs-agentd/config.ini", "644"],
     ["sonic/tacacsrs-agentd.service", "lib/systemd/system/tacacsrs-agentd.service", "644"],
     ["../../docs/tacacsrs-agentd.md", "usr/share/doc/tacacsrs-agentd/README.md", "644"],

executables/tacacsrs_agentd/README.md

@@ -4,6 +4,21 @@
 
 For full deployment and failover guidance, see [../../docs/tacacsrs-agentd.md](../../docs/tacacsrs-agentd.md).
 
+## Health and Host Integration
+
+When the Client API is enabled, its existing endpoint also serves standard `grpc.health.v1.Health` names for startup, liveness, readiness, the overall empty name, and `tacacsrs.agent.v1.TacacsAgent`. Use the packaged probe:
+
+```bash
+tacacsrs-agent-health \
+    --endpoint /run/tacacs/tacacs.sock \
+    --check readiness \
+    --timeout-seconds 2
+```
+
+Use `--host-integration none` in containers. `auto` selects systemd only when `NOTIFY_SOCKET` is present. Explicit `systemd` requires the notification socket and helper, sends `READY=1` once, and sends `STOPPING=1` before listener drain.
+
+SONiC ConfigDB startup and notification subscriptions are supervised. The daemon may bind listeners before Redis exists, retries with capped jittered backoff, applies valid snapshots without restarting, and retains the previous known-good configuration after invalid reloads or subscription outages.
+
 ## Runtime Service Modes
 
 Use `--service-mode client-api`, `--service-mode tacacs-proxy`, or

executables/tacacsrs_agentd/sonic/tacacsrs-agentd.service

@@ -1,11 +1,10 @@
 [Unit]
 Description=Central TACACS+ client service for local consumers (SONiC)
 Documentation=https://github.com/AuthScaffold/tacacs-rs
-# SONiC stores configuration in Redis (CONFIG_DB). Wait for the database
-# service so that ConfigDB is ready before we attempt to read TACPLUS tables.
-Requires=database.service
+# SONiC stores configuration in Redis (CONFIG_DB). Ordering normally avoids an
+# initial retry, but the daemon also survives database startup and restarts.
 After=database.service network-online.target
-Wants=network-online.target
+Wants=database.service network-online.target
 
 [Service]
 # Keep the unit in "activating" with a clear status line until SONiC pushes
@@ -17,6 +16,7 @@ ExecStart=/usr/sbin/tacacsrs-agentd \
     --sonic \
     --listen-endpoint /run/tacacs/tacacs.sock \
     --socket-mode 666 \
+    --host-integration systemd \
     -vv
 
 # Recreate the socket after restart and tear it down on stop.
@@ -28,8 +28,8 @@ TimeoutStartSec=0
 TimeoutStopSec=15s
 
 # SONiC keyspace notifications must be enabled on Redis for hot reload to
-# work; the agent will start without them but will only react to changes on
-# restart.
+# work. The agent retries subscription setup and reloads before resubscribing
+# after a notification stream ends.
 #   redis-cli -n 4 CONFIG SET notify-keyspace-events KEA
 
 # Hardening (loose defaults compatible with SONiC's privileged service model;