Hosting Integration and Health Model

[Rod-Persky]

This pull request introduces several important improvements to the tacacsrs-agentd project, focusing on enhanced health checking, improved host integration, and more robust SONiC ConfigDB handling. The changes include a new health probe executable, expanded documentation, and updates to deployment/service files to support systemd notification and containerized environments.

Health checking and host integration:

• Added a new tacacsrs-agent-health executable for checking the agent's gRPC health endpoints, with clear exit codes for use in Kubernetes exec probes and diagnostics (executables/tacacsrs_agentd/src/bin/tacacsrs-agent-health.rs, Cargo.toml, README.md). [1] [2] [3]
• Introduced the --host-integration flag to control systemd notification, with modes for auto, systemd, and none (container-friendly). Documentation and service files updated accordingly (docs/tacacsrs-agentd.md, sonic-agentd-container.md, sonic/tacacsrs-agentd.service). [1] [2] [3]

SONiC and ConfigDB improvements:

• The agent now starts and binds listeners even if Redis/ConfigDB is temporarily unavailable, marking readiness as not serving until a valid snapshot is loaded. Subscription failures and invalid reloads are handled gracefully, with automatic retries and health status updates (docs/sonic-agentd-container.md, sonic-configdb-integration.md, tacacsrs-agentd.md). [1] [2] [3]

Shared secret and proxy mode behavior:

• Documented and clarified that downstream TACACS+ obfuscation is now independent of the selected upstream server, with SONiC mode using the local loopback proxy row's secret and non-SONiC mode using --proxy-shared-secret (tacacsrs-agentd.md, tacacs-plus-tls-transition.md). [1] [2]

Packaging and deployment:

• Updated packaging to include the new health probe executable and adjusted systemd service files to use Type=notify and enable systemd notification integration (Cargo.toml, tacacsrs-agentd.service). [1] [2] [3]

Dependency updates:

• Added new dependencies for async traits, random backoff, health checking, and streaming utilities to support the new features (Cargo.toml). [1] [2]

These changes improve observability, reliability, and operational flexibility for both containerized and systemd-based deployments.

[Copilot]

Pull request overview

This pull request adds a protocol-neutral runtime health model to tacacsrs-agentd and associated crates, then uses it to drive standard gRPC health checks, supervised SONiC ConfigDB loading/reloading, and optional host integration (systemd notifications) suitable for both systemd units and container deployments.

Changes:

• Introduces typed runtime health publication (watch-backed snapshots), listener lifecycle registration, and coordinated shutdown/drain semantics in tacacsrs-agent.
• Updates datastore contracts to include declared runtime policy (initial-load + notification mode) and to emit typed change events (changed vs candidate rejected), enabling resilient ConfigDB/file reload supervision.
• Adds a standalone tacacsrs-agent-health executable plus deployment/docs updates for exec probes, systemd Type=notify, and container-friendly operation.

Reviewed changes

Copilot reviewed 49 out of 50 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
libraries/tacacsrs_sonic/src/mapping.rs	Uses shared TacacsPlus::empty() helper for empty SONiC table mapping.
libraries/tacacsrs_sonic/src/lib.rs	Implements new datastore runtime policy + typed change stream for SONiC ConfigDB.
libraries/tacacsrs_sonic/examples/configdb_watch.rs	Updates example to handle ConfigChangeEvent variants.
libraries/tacacsrs_datastore/src/lib.rs	Introduces ConfigChangeEvent + datastore runtime policy contract and updates helpers/tests.
libraries/tacacsrs_config/src/builders.rs	Adds TacacsPlus::empty() and reuses it in the builder.
libraries/tacacsrs_config/Cargo.toml	Bumps base64 dependency.
libraries/tacacsrs_cli_datastore/src/datastore.rs	Implements runtime policy and emits typed change events (including candidate rejection).
libraries/tacacsrs_cli_datastore/Cargo.toml	Bumps base64 dependency.
libraries/tacacsrs_agent/src/upstream/manager/mod.rs	Wires upstream availability tracking into runtime health and resets on material reloads.
libraries/tacacsrs_agent/src/upstream/manager/availability.rs	Adds race-safe aggregate upstream availability publisher + tests.
libraries/tacacsrs_agent/src/services/tacacs_proxy/service.rs	Passes shutdown + listener registration into proxy listener serving.
libraries/tacacsrs_agent/src/services/tacacs_proxy/listener/unix.rs	Marks listener bound in health and respects shared shutdown receiver.
libraries/tacacsrs_agent/src/services/tacacs_proxy/listener/tcp.rs	Marks listener bound in health and respects shared shutdown receiver.
libraries/tacacsrs_agent/src/services/tacacs_proxy/listener/mod.rs	Refactors accept loop to use ShutdownReceiver and listener registration.
libraries/tacacsrs_agent/src/services/client_api/upstream_bridge/mod.rs	Updates tests to construct upstream manager with health publisher.
libraries/tacacsrs_agent/src/services/client_api/service.rs	Adds shutdown + health integration and provides health snapshot receiver to listener.
libraries/tacacsrs_agent/src/services/client_api/mod.rs	Introduces client_api::health module.
libraries/tacacsrs_agent/src/services/client_api/listener/unix.rs	Serves standard gRPC health service and coordinates shutdown/drain.
libraries/tacacsrs_agent/src/services/client_api/listener/tcp.rs	Serves standard gRPC health service and coordinates shutdown/drain.
libraries/tacacsrs_agent/src/services/client_api/listener/mod.rs	Threads shutdown/registration/health receiver through listener selection.
libraries/tacacsrs_agent/src/services/client_api/health.rs	Implements standard grpc.health.v1.Health mapping from runtime snapshots + tests.
libraries/tacacsrs_agent/src/runtime/shutdown.rs	Adds shared shutdown coordination + listener registration guard + tests.
libraries/tacacsrs_agent/src/runtime/mod.rs	Re-exports new runtime health/shutdown types.
libraries/tacacsrs_agent/src/runtime/health.rs	Adds typed health model, publisher, readiness/liveness/startup semantics + tests.
libraries/tacacsrs_agent/src/runtime/client_service.rs	Integrates health + shutdown coordinator into serving lifecycle and reload behavior.
libraries/tacacsrs_agent/src/lib.rs	Re-exports runtime health types publicly.
libraries/tacacsrs_agent/README.md	Updates runtime lifecycle/shutdown and health model documentation.
libraries/tacacsrs_agent/Cargo.toml	Adds tonic-health and test-util tokio dev-dependency.
libraries/tacacsrs_agent_ipc_emulator/Cargo.toml	Updates regorus dependency version.
libraries/tacacsrs_agent_client/src/lib.rs	Exposes new health module and re-exports HealthClient.
libraries/tacacsrs_agent_client/src/health.rs	Adds stable health service names and a standard gRPC health client.
libraries/tacacsrs_agent_client/src/endpoint.rs	Extracts reusable connect_channel() for both service + health clients.
libraries/tacacsrs_agent_client/src/client.rs	Refactors ServiceClient to reuse shared channel connection helper.
libraries/tacacsrs_agent_client/Cargo.toml	Adds tonic-health dependency.
lde/run-p1-health-process-smoke.sh	Adds a process-level smoke script covering UDS replacement, pre-Redis startup, shutdown cleanup, and health recovery.
executables/tacacsrs_agentd/tests/health_probe.rs	Adds integration tests for probe exit codes and health protocol behavior.
executables/tacacsrs_agentd/src/systemd_notify.rs	Removes legacy systemd notification shim (replaced by host integration).
executables/tacacsrs_agentd/src/main.rs	Introduces config supervision + host integration and starts service in “waiting for config” mode.
executables/tacacsrs_agentd/src/host_integration.rs	Adds --host-integration handling and systemd notification bridge from health snapshots.
executables/tacacsrs_agentd/src/config_supervisor.rs	Adds cancellable load/retry/subscribe supervision with backoff and health state updates.
executables/tacacsrs_agentd/src/cli.rs	Adds --host-integration flag and HostIntegrationMode enum.
executables/tacacsrs_agentd/src/bin/tacacsrs-agent-health.rs	Adds tacacsrs-agent-health exec probe binary with stable exit codes.
executables/tacacsrs_agentd/sonic/tacacsrs-agentd.service	Updates SONiC unit dependencies and enables strict systemd host integration.
executables/tacacsrs_agentd/README.md	Documents health endpoint/probe usage and host integration modes.
executables/tacacsrs_agentd/Cargo.toml	Adds deps for supervision/health and packages the new probe executable.
docs/tacacsrs-agentd.md	Documents health model, probe contract, supervised ConfigDB behavior, and systemd notify settings.
docs/tacacs-plus-tls-transition.md	Updates downstream shared-secret guidance for proxy mode under new behavior.
docs/sonic-configdb-integration.md	Updates SONiC hot-reload behavior to describe atomic in-process reloads and retry semantics.
docs/sonic-agentd-container.md	Updates container deployment guidance (mounts, host integration none, and exec probe usage).
Cargo.lock	Updates lockfile for new/bumped dependencies (including tonic-health, base64, etc.).

[Copilot]

Pull request overview

Copilot reviewed 49 out of 50 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (1)

executables/tacacsrs_agentd/src/host_integration.rs:153

• When systemd-notify fails in non-strict (auto) mode, the warning log drops the underlying error, which makes diagnosing why notifications aren’t being published unnecessarily hard (e.g., missing helper vs. exit status vs. execution failure). Include the error details in the log message (it should not contain secrets).

        if let Err(error) = self.command.execute(&arguments) {
            if self.strict {
                return Err(error);
            }
            log::warn!("Failed to publish automatic systemd notification");
        }

[Copilot]

Pull request overview

Copilot reviewed 51 out of 52 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (2)

executables/tacacsrs_agentd/src/host_integration.rs:120

• The tokio::select! is marked biased; with the health.changed() branch first. If health updates arrive continuously, this can starve the cancellation branch and delay (or potentially prevent) host integration shutdown, especially during noisy health transitions.

                    tokio::select! {
                        biased;
                        result = health.changed() => {
                            if result.is_err() {
                                return Ok(());

executables/tacacsrs_agentd/src/host_integration.rs:152

• In auto mode, notification failures are intentionally non-fatal, but the warning currently drops the underlying error. Including the error context here will make diagnosing missing systemd-notify or permission issues much easier.

        if let Err(error) = self.command.execute(&arguments) {
            if self.strict {
                return Err(error);
            }
            log::warn!("Failed to publish automatic systemd notification");