configure.md

Configure Sōzu

Before a deep dive in the configuration part of the proxy, you should take a look at the getting started documentation if you haven't yet.

Configuration file

The configuration file uses the .toml format.

Sōzu configuration process involves 3 major sources of parameters:

• The global section, which sets process-wide parameters.
• The definition of the protocols like https, http, tcp.
• The clusters sections under: [clusters].

Global parameters

Parameters in the global section allow you to define the global settings shared by the main process and workers (like the log level):

parameter	description	possible values
saved_state	path from which sozu tries to load its state at startup
automatic_state_save	if saved_state is set, persists state to it whenever a command changes routing. Defaults to false	true, false
log_level	possible values are	debug, trace, error, warn, info
log_target	possible values are	stdout, tcp or udp address
log_colored	emit ANSI colour codes on the main log stream. Only honoured when log_target is stdout (tcp / udp / file sinks ignore this flag). Defaults to false	true, false
access_logs_target	possible values are (if activated, sends access logs to a separate target)	stdout, tcp or udp address
access_logs_format	wire format of access logs. Defaults to ascii	ascii, protobuf
access_logs_colored	emit ANSI colour codes on access logs. Only honoured when the access-log target is stdout. If unset, inherits log_colored. Defaults to false	true, false
command_socket	path to the unix socket command
command_buffer_size	size, in bytes, of the buffer used by the main process to handle commands.
max_command_buffer_size	maximum size of the buffer used by the main process to handle commands.
worker_count	number of workers
worker_automatic_restart	if activated, workers that panicked or crashed are restarted (activated by default)
worker_timeout	maximum time (in seconds) the main process waits for a worker reply before marking it NotAnswering. Defaults to 10	seconds
disable_cluster_metrics	if true, per-cluster metrics are not registered. Defaults to false (cluster metrics enabled)	true, false
handle_process_affinity	bind workers to cpu cores.
max_connections	maximum number of simultaneous / opened connections
max_buffers	maximum number of buffers use to proxying
min_buffers	minimum number of buffers preallocated for proxying
buffer_size	size, in bytes, of requests buffer used by the workers. Must be at least 16393 for HTTP/2 (16384 max frame size + 9 byte frame header)
slab_entries_per_connection	how many slab entries each max_connections reserves. Defaults to 4 (1 frontend + up to 3 backend H2 connections). Raise for fan-out topologies that exceed 4 backends per session; clamped to [2, 32]. Slab capacity is 10 + slab_entries_per_connection * max_connections.	integer 2-32
splice_pipe_capacity_bytes	requested kernel-pipe capacity, in bytes, per splice(2) direction on Protocol::TCP listeners (Linux only, requires the splice cargo feature). Omitted or None keeps the kernel default of 64 KiB. Applied via fcntl(F_SETPIPE_SZ) per pipe at session start; the kernel rounds up to a page boundary and clamps at /proc/sys/fs/pipe-max-size (default 1 MiB unprivileged; CAP_SYS_RESOURCE goes higher). The realised capacity is read back via fcntl(F_GETPIPE_SZ) and used as the per-call len for splice_in. Larger values amortise syscalls and reduce wakeups for bulk-transfer workloads at the cost of per-session pinned memory; raise /proc/sys/fs/pipe-max-size first if you want above 1 MiB without root.	integer (bytes), e.g. 262144
command_allowed_uids	optional allowlist of POSIX UIDs permitted to invoke command-socket requests (PR #1209). Omitted or None preserves the historical "any same-UID local process" behaviour. Set to [<operator_uid>] to restrict mutating verbs to a specific UID even when other same-UID daemons coexist (CI runners, monitoring scrapers). Rejected requests still appear in the audit trail.	TOML array of integers, e.g. [1000]
ctl_command_timeout	maximum time the command line will wait for a command to complete
pid_file_path	stores the pid in a specific file location
front_timeout	maximum time of inactivity for a front socket
back_timeout	maximum time of inactivity for a backend socket (seconds). Defaults to 30. Can be overridden per listener.	seconds
connect_timeout	maximum time of inactivity for a request to connect
accept_queue_timeout	maximum time (in seconds) a TCP connection stays in sozu's accept queue before being dropped. Defaults to 60.	seconds
request_timeout	maximum time of inactivity for a request
zombie_check_interval	duration between checks for zombie sessions
evict_on_queue_full	evict the least-recently-active sessions when max_connections is reached, making room for new accepts. Defaults to false: during a DDoS the existing connections are more likely to be legitimate clients than the queued ones, so refusing new accepts is the safer mitigation. Enable when overload is dominated by normal traffic spikes. Triggers a config-load warn! when max_connections < 100 because the 1% eviction batch clamps to 1 (so the per-round share grows).	true, false (default: false)
activate_listeners	automatically start listeners
max_connections_per_ip	global default per-(cluster, source-IP) connection limit. 0 disables the feature (default). The source IP is taken from the parsed PROXY-protocol header when present, else peer_addr. HTTP/HTTPS clients hitting the limit receive 429 Too Many Requests; TCP clients see a graceful FIN. Each cluster may override via its own max_connections_per_ip field (None inherits, Some(0) is explicit unlimited, Some(n > 0) overrides). Counters are kept per (cluster_id, source_ip), so two clusters never share a counter.	integer (0 = unlimited)
retry_after	global default Retry-After header value (seconds) for HTTP 429 responses. 0 omits the header — Retry-After: 0 invites an immediate retry that defeats the limit. Each cluster may override. TCP listeners ignore this value (no HTTP envelope).	integer (0 = omit)

Example:

command_socket = "./command_folder/sock"
saved_state = "./state.json"
log_level = "info"
log_target = "stdout"
command_buffer_size = 16384
worker_count = 2
handle_process_affinity = false
max_connections = 500
max_buffers = 500
buffer_size = 16393
activate_listeners = true

Privacy note on logs and retention

log_target at debug/trace emits per-connection context on every protocol log line, including:

• peer IP (client source address)
• SNI (TLS hostname the client requested) on TLS listeners
• a per-session ULID generated at connection accept
• mio frontend token and cluster/backend identifiers

Each of these is an identifier under most data-protection regimes (GDPR, CNIL guidance on proxy logs). Keep production workers at log_level = "info" or tighter unless debugging; access logs (access_logs_target) carry the same fields in a shape meant for long-term retention and should be the durable store.

Retention of the live log stream depends on the sink:

• stdout: inherited from the surrounding process supervisor (journald, the init system, the container runtime). On Clever Cloud ADCs the journald cap is size-bounded at ~4 GB — time coverage varies with traffic.
• tcp:// / udp://: forwarded to the remote collector; retention becomes the collector's responsibility. Confirm DPA coverage before routing logs to a third party.

If logs egress beyond Clever Cloud infrastructure, the per-session ULID plus peer IP plus SNI combination is a durable cross-system correlator — treat it accordingly in your privacy impact assessment.

Listeners

The listener section describes a set of listening sockets accepting client connections. You can define as many listeners as you want. They follow the format:

General parameters:

[[listeners]]
# possible values are http, https, tcp or udp
protocol = "http"
# listening address
address = "0.0.0.0:8080"
# address = "[::]:8080"

# specify a different IP than the one the socket sees, for logs and forwarded headers
# public_address = "1.2.3.4:80

# Configures the client socket to receive a PROXY protocol header
# (TCP listeners only — not supported on UDP listeners)
# expect_proxy = false

Options specific to HTTP and HTTPS listeners

Since version 1.0.0, Sōzu allows custom HTTP answers defined for HTTP and HTTPS listeners.

These answers are customizable:

• 301 Moved Permanently
• 400 Bad Request
• 401 Unauthorized
• 404 Not Found
• 408 Request Timeout
• 413 Payload Too Large
• 502 Bad Gateway
• 503 Service Unavailable
• 504 Gateway Timeout
• 507 Insufficient Storage

These answers are to be provided in plain text files of whichever extension (we recommend .http for clarity) and may look like this:

HTTP/1.1 404 Not Found Cache-Control: no-cache Connection: close Sozu-Id:
%REQUEST_ID

<style>
  pre {
    background: #eee;
    padding: 10px;
    border: 1px solid #aaa;
    border-radius: 5px;
  }
</style>
<h1>404 Not Found</h1>

<p>
  insert your custom text here, in fact, all HTML is changeable, including the
  CSS.
</p>

<pre>
{
    \"route\": \"%ROUTE\",
    \"request_id\": \"%REQUEST_ID\",
}
</pre>
<footer>This is an automatic answer by Sozu.</footer>
",

There are a number of available template variables, like REQUEST_ID or CLUSTER_ID, that will be replaced by the proxying logic when producing the error.

To create your own custom HTTP answers, we highly suggest you first copy the default answers present in lib/src/protocol/kawa_h1/answers.rs, and then change them to your liking. Feel free to remove the \r newlines of the default strings for clarity. Sōzu will parse your file and replace whatever newline symbol(s) you use.

Then, for each listener, provide the absolute paths of each custom answer.

# a 404 response is sent when sozu does not know about the requested domain or path
answer_404 = "/path/to/my-404-answer.http"
# a 503 response is sent if there are no backend servers available
answer_503 = "/path/to/my-503-answer.http"
# answer_507 = ...

If a frontend has a sticky_session, the sticky name is defined at the listener level.

# defines the sticky session cookie's name, if `sticky_session` is activated format
# a cluster. Defaults to "SOZUBALANCEID"
sticky_name = "SOZUBALANCEID"

Options specific to HTTPS listeners

Crypto provider: The cryptographic backend used by Rustls is a compile-time choice (feature flags crypto-ring, crypto-aws-lc-rs, crypto-openssl, fips). It cannot be changed at runtime. See Getting started — Choosing a crypto provider for build instructions.

# supported TLS versions. Possible values are "SSL_V2", "SSL_V3",
# "TLS_V12", "TLS_V13". Defaults to "TLS_V12" and "TLS_V13"
tls_versions = ["TLS_V12", "TLS_V13"]

HSTS — HTTP Strict Transport Security (RFC 6797)

HSTS = HTTP Strict Transport Security (RFC 6797). When configured on an HTTPS listener or frontend, Sōzu emits the Strict-Transport-Security response header so conformant browsers refuse to talk to the host over plaintext HTTP for the configured max-age duration.

A [hsts] block under an HTTPS listener is the operator-default; per-frontend overrides live under [clusters.<id>.frontends.hsts]. Per-frontend enabled = true overrides the listener default; per-frontend enabled = false explicitly suppresses an inherited listener default for that frontend.

Per RFC 6797 §7.2 the Strict-Transport-Security header MUST NOT appear on plaintext-HTTP responses. Sōzu enforces this in three layers: (1) TOML config-load rejects [hsts] on a plain-HTTP listener (ConfigError::HstsOnPlainHttp), (2) the worker IPC entry rejects AddHttpFrontend carrying an enabled HSTS (ProxyError::HstsOnPlainHttp and the http.hsts.suppressed_plaintext counter), and (3) the runtime per-stream snapshot copy is gated on context.protocol == Protocol::HTTPS so plaintext connections never apply HSTS edits even when one slips through.

Is the [hsts] block required?

No — it is optional everywhere. Omitting the block on every listener and every frontend is a fully valid configuration: no Strict-Transport-Security header is ever emitted.

Surface	Required?	Notes
[hsts] under a plain-HTTP listener / frontend	N/A	The field does not exist on the HTTP listener proto. Setting it on an HTTP frontend is rejected at config-load (RFC §7.2).
[hsts] under an HTTPS listener (operator default)	Optional	Omit the block and no listener-default HSTS is set. HTTPS frontends without their own block then carry no HSTS.
[hsts] under a per-frontend section	Optional	Omit and the frontend inherits the listener default (or nothing if the listener has none).
[hsts] is not a cluster-level field	N/A	HSTS lives on frontends (Caddy / Traefik shape), not on clusters.
UpdateHttpsListenerConfig.hsts (hot-reconfig partial patch)	Optional	Absent in the patch preserves the current listener HSTS value. Present = full-object replacement (see below).

The only conditional required-field rule is inside the block itself: when the [hsts] block is present (TOML or partial-update patch), the enabled field must also be present. This disambiguates three otherwise-conflated semantics:

TOML / patch shape	Meaning
Block omitted	Inherit listener default (or no HSTS if the listener has none).
[hsts] with enabled = true	Explicitly enable HSTS at this scope. max_age defaults to 31_536_000 if omitted.
[hsts] with enabled = false	Explicitly disable HSTS at this scope, suppressing any inherited listener default.
[hsts] without enabled	Error — ConfigError::HstsEnabledRequired (TOML) or ListenerError::HstsEnabledRequired (partial update).

A minimal HTTPS deployment with no HSTS is therefore just:

[[listeners]]
protocol = "https"
address  = "0.0.0.0:443"
# no [hsts] block — fine

[[clusters.api.frontends]]
address  = "0.0.0.0:443"
hostname = "api.example.com"
# no [hsts] block — fine

# Listener-level default — every HTTPS frontend on this listener
# inherits unless it declares its own [hsts] block. The `[hsts]`
# table nests under the enclosing `[[listeners]]` entry.
[[listeners]]
protocol = "https"
address  = "0.0.0.0:443"

[hsts]
# REQUIRED whenever the [hsts] block is present. `false` is the
# explicit-disable signal on a partial-update; new TOML deployments
# will normally set `true`.
enabled = true
# `Strict-Transport-Security: max-age=<seconds>`. When omitted with
# `enabled = true`, sozu substitutes 31_536_000 seconds (1 year — the
# Chrome HSTS preload list minimum) at config-load.
max_age = 31536000
# Append `; includeSubDomains` to the rendered header.
include_subdomains = true
# Append `; preload` to the rendered header. Opt-in only — once
# submitted to https://hstspreload.org/, removal is slow and partial
# (RFC 6797 §14.2). NEVER default-true.
preload = false

Per-frontend override or explicit disable:

[[clusters.api.frontends]]
address  = "0.0.0.0:443"
hostname = "api.example.com"

# Override the listener default with a longer 2-year max-age and
# opt the host into the preload list.
[clusters.api.frontends.hsts]
enabled            = true
max_age            = 63072000
include_subdomains = true
preload            = true

# Suppress the inherited listener default for a legacy frontend that
# cannot commit to HSTS yet:
[[clusters.legacy.frontends]]
address  = "0.0.0.0:443"
hostname = "legacy.example.com"

[clusters.legacy.frontends.hsts]
enabled = false

Validation matrix

Configuration	Outcome
[hsts] block without enabled	Error HstsEnabledRequired at config-load. enabled is the explicit-disambiguator between disable / enable on partial updates.
[hsts] on a plain-HTTP listener or frontend	Error HstsOnPlainHttp at config-load (and ProxyError::HstsOnPlainHttp if it slipped through to the worker). RFC 6797 §7.2.
enabled = true, max_age omitted	Substituted to DEFAULT_HSTS_MAX_AGE = 31_536_000 (1 year). Matches the HSTS preload list minimum.
max_age = 0	Allowed silently — RFC 6797 §11.4 kill switch. Conformant UAs stop treating the host as a Known HSTS Host.
0 < max_age < 86_400	Warning at config-load (likely misconfiguration — sub-day HSTS only makes sense for testing).
preload = true with max_age < 31_536_000	Warning at config-load. The Chrome HSTS preload list rejects hosts below the minimum.
preload = true without include_subdomains = true	Warning at config-load. Same preload-list rejection.
Backend emits its own Strict-Transport-Security	Pass-through unchanged by default. Sōzu's HSTS edit uses HeaderEditMode::SetIfAbsent so a single header reaches the wire (RFC 6797 §6.1). Set force_replace_backend = true to override the backend value with sōzu's typed policy (HeaderEditMode::Set — delete-then-insert).
HTTPS-served default answer (3xx redirect, 401, 503)	Carries the HSTS header per RFC 6797 §8.1. The per-stream snapshot copy fires before the early returns in mux/router.rs, gated on Protocol::HTTPS.

Runtime CLI surface

Per-frontend override (replaces the listener default for one frontend):

sozu frontend https add \
  --address 0.0.0.0:443 \
  --hostname api.example.com \
  --hsts-max-age 31536000 \
  --hsts-include-subdomains

Listener-default patch (replaces the policy for every frontend that inherits from the listener):

sozu listener https update \
  -a 0.0.0.0:443 \
  --hsts-max-age 31536000 \
  --hsts-include-subdomains

Both surfaces share the same flag set: --hsts-max-age, --hsts-include-subdomains, --hsts-preload, --hsts-force-replace-backend, and the kill-switch --hsts-disabled. The latter is mutually exclusive with the four enabling flags on either path; combining them surfaces CtlError::ArgsNeeded from the shared build_hsts_from_cli helper rather than a silent pick.

Hot-reconfig partial-update

UpdateHttpsListenerConfig.hsts follows full-object replacement semantics: when present in the patch, the entire HSTS block replaces the listener's current value. enabled is REQUIRED whenever hsts is present (ListenerError::HstsEnabledRequired rejects an enabled = None block). Absent hsts field on the patch preserves the current value. The CLI surface above (sozu listener https update --hsts-*) feeds this same partial-update message; supplying any --hsts-* flag on the command line replaces the listener's HSTS policy and supplying --hsts-disabled substitutes the explicit-disable block (enabled = Some(false)).

Inheriting frontends are refreshed automatically. Patching the listener-default HSTS reflows the new policy onto every existing frontend that inherited from the listener (i.e. has no per-frontend [hsts] block at add time). Router::refresh_inheriting_hsts walks the routing trie via two paths:

1. Existing Route::Frontend entries with inherits_listener_hsts == true have their headers_response rebuilt — any prior Strict-Transport-Security entry is stripped and a freshly rendered one appended.
2. Lightweight Route::ClusterId and Route::Deny entries — the routing fast path stores frontends without per-frontend policy fields (no redirect, no rewrite, no headers, no auth, no HSTS) under these compact variants, NOT as Route::Frontend — are promoted in place to a minimal Route::Frontend carrying just the HSTS edit on headers_response. The promoted entry is marked inherits_listener_hsts = true so subsequent patches refresh it through path 1. Routing semantics are preserved (the promoted Frontend forwards / denies identically to the original variant). When the new policy resolves to "no HSTS" (None / disabled) the lightweight routes are left untouched — no allocation is created just to hold an empty edit.

Per-frontend explicit overrides (inherits_listener_hsts == false on path 1) are never touched. The http.hsts.listener_default_patched counter fires once per patch and http.hsts.frontend_refreshed increments per refreshed frontend (path 1 + path 2 promotions combined), so dashboards can correlate the rate of patches with the size of the affected fleet.

Path 2 was added after the initial HSTS rollout to fix a silent skip that affected fleets where most frontends are added without per-frontend policy fields (e.g. Clever Cloud cleverapps.io shared, where 91 k+ frontends were stored as Route::ClusterId and never received the listener-default HSTS edit).

Metrics

Metric	Kind	Emitted when
http.hsts.frontend_added	counter	A frontend is added with hsts.enabled = true and the HSTS edit is materialised in headers_response.
http.hsts.suppressed_plaintext	counter	An AddHttpFrontend IPC was rejected because it carried an enabled HSTS policy (RFC 6797 §7.2 defense in depth).
http.hsts.unrendered	counter	Defense-in-depth: a frontend reached Frontend::new with hsts.enabled = true but render_hsts returned None (max_age missing). The TOML loader and the CLI helper both substitute DEFAULT_HSTS_MAX_AGE so this counter only fires when a programmatic IPC sender ships an ill-formed HstsConfig.
http.hsts.listener_default_patched	counter	A UpdateHttpsListenerConfig.hsts patch was applied. Fires once per patch.
http.hsts.frontend_refreshed	counter	An inheriting frontend was refreshed during a listener-default HSTS patch (one increment per refreshed entry). Sum across a patch interval = number of frontends touched by that patch.

Options specific to Rustls based HTTPS listeners

Cipher suites

# Sets the list of available cipher suites, in order of preference.
# If omitted, the following default list is used (ANSSI-recommended order):
cipher_list = [
    # TLS 1.3 cipher suites
    "TLS13_AES_256_GCM_SHA384",
    "TLS13_AES_128_GCM_SHA256",
    "TLS13_CHACHA20_POLY1305_SHA256",
    # TLS 1.2 cipher suites
    "TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384",
    "TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256",
    "TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256",
    "TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384",
    "TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256",
    "TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256",
]

HTTP/2 support and ALPN protocols

HTTP/2 is available on HTTPS listeners through ALPN (Application-Layer Protocol Negotiation). During the TLS handshake, Sōzu advertises protocols from the alpn_protocols list. The server selects the first protocol from its list that the client also supports.

By default, Sōzu advertises both h2 and http/1.1, preferring HTTP/2:

# Default: both protocols, H2 preferred
alpn_protocols = ["h2", "http/1.1"]

Value	Protocol	Notes
h2	HTTP/2	Multiplexed, binary framing (RFC 9113)
http/1.1	HTTP/1.1	Traditional text-based protocol (RFC 9112)

Invalid values are rejected at configuration load time. Order matters: the first entry is the most preferred protocol.

Examples:

# HTTP/1.1 only — disables HTTP/2 on this listener
alpn_protocols = ["http/1.1"]

# HTTP/2 only — clients without H2 support will fail TLS negotiation
alpn_protocols = ["h2"]

# Prefer HTTP/1.1 over HTTP/2
alpn_protocols = ["http/1.1", "h2"]

When alpn_protocols is omitted or empty, the default ["h2", "http/1.1"] is used. Clients that do not send an ALPN extension default to HTTP/1.1.

Note: HTTP/2 is only supported over TLS (HTTPS listeners). Plain HTTP listeners always use HTTP/1.1.

Note: When using the fips crypto provider, CHACHA20_POLY1305 cipher suites are not available. Only AES-GCM suites are FIPS-approved. Sōzu's cipher_suite_by_name filters its result through the active provider's supported set so a misconfigured cipher_list cannot silently downgrade an FIPS build.

Key exchange groups

The groups_list option controls which key exchange algorithms are offered during the TLS handshake. Groups are listed in order of preference.

# Default: ["X25519MLKEM768", "x25519", "P-256", "P-384"]
groups_list = ["X25519MLKEM768", "x25519", "P-256", "P-384"]

Group name	Description	Provider support
x25519 / X25519	Curve25519 ECDHE	All providers
secp256r1 / P-256	NIST P-256 ECDHE	All providers
secp384r1 / P-384	NIST P-384 ECDHE	All providers
X25519MLKEM768	Post-quantum hybrid (X25519 + ML-KEM 768)	crypto-aws-lc-rs, crypto-openssl (OpenSSL 3.5+)

Unknown or unsupported group names are silently skipped with a log warning. This allows using the same configuration across different crypto providers — for example, X25519MLKEM768 is safely ignored when building with crypto-ring.

Examples:

# Post-quantum enabled (default). PQ-capable clients negotiate X25519MLKEM768,
# others fall back to classical X25519.
groups_list = ["X25519MLKEM768", "x25519", "P-256", "P-384"]

# Classical only (explicitly disable post-quantum)
groups_list = ["x25519", "P-256", "P-384"]

# FIPS 140-3 compliant (NIST curves only, no X25519)
groups_list = ["P-256", "P-384"]

Post-quantum key exchange: X25519MLKEM768 is a hybrid scheme that combines classical X25519 with the ML-KEM 768 post-quantum algorithm. It protects against future quantum computer attacks while maintaining security against current classical attacks. The handshake is slightly larger (~1 KB overhead) but has negligible latency impact. Clients that do not support it automatically negotiate a classical group.

Certificates

TLS certificates can be configured in two ways:

1. Default certificate on the HTTPS listener (without SNI):

[[listeners]]
protocol = "https"
address = "0.0.0.0:8443"
certificate = "/path/to/certificate.pem"
certificate_chain = "/path/to/chain.pem"
key = "/path/to/private-key.pem"

2. Per-frontend certificates (with SNI, recommended):

[clusters.MyCluster]
protocol = "http"
frontends = [
    { address = "0.0.0.0:8443", hostname = "example.com",
      certificate = "/path/to/example.com.pem",
      certificate_chain = "/path/to/chain.pem",
      key = "/path/to/example.com.key" },
]
backends = [
    { address = "127.0.0.1:8080" }
]

Sōzu supports the following certificate and key types:

Type	Key format	Notes
RSA 2048+	PKCS#1 or PKCS#8 PEM	Most common, widely supported
ECDSA P-256	SEC1 or PKCS#8 PEM	Faster handshakes, smaller certificates
ECDSA P-384	SEC1 or PKCS#8 PEM	Higher security margin

All certificate files must be PEM-encoded. The certificate_chain should contain intermediate CA certificates (not the root CA).

Generating test certificates:

# RSA 2048
openssl req -newkey rsa:2048 -nodes -keyout rsa.key -x509 -days 365 \
    -subj "/CN=example.com" -addext "subjectAltName=DNS:example.com" -out rsa.pem

# ECDSA P-256
openssl ecparam -name prime256v1 -genkey -out ecdsa.key
openssl req -new -key ecdsa.key -x509 -days 365 \
    -subj "/CN=example.com" -addext "subjectAltName=DNS:example.com" -out ecdsa.pem

Important: Certificates must include a Subject Alternative Name (SAN) extension matching the frontend hostname. Certificates without SANs may cause TLS handshake failures.

Options specific to TCP listeners

A protocol = "tcp" listener forwards raw bytes to a cluster's backends without terminating TLS. By default one TCP listener routes to exactly one cluster (the legacy shape). Since #1279 a TCP listener can instead fan out to multiple clusters on the same address:port by reading the TLS ClientHello's SNI (RFC 6066 §3) and, per route entry, ALPN (RFC 7301 §3.1) — all without decrypting or terminating the connection. The backend still performs its own TLS handshake with the unmodified client bytes.

[[listeners]]
protocol = "tcp"
address  = "0.0.0.0:8443"

# Time allowed to receive enough bytes of the TLS ClientHello to read the SNI
# extension, in seconds. Only meaningful once at least one SNI-scoped
# frontend (see below) targets this listener — a listener with only
# no-SNI (legacy) frontends never prereads and ignores this value.
# Defaults to 5. Must not exceed this listener's front_timeout
# (rejected at TOML config-load; see the validation matrix below).
sni_preread_timeout = 5

# Maximum bytes buffered while prereading the ClientHello looking for the SNI
# extension. Only meaningful under the same condition as above. Clamped to
# the global buffer_size at runtime (>= sni_preread_max_bytes is required at
# TOML config-load; see the validation matrix below). Defaults to 16384.
sni_preread_max_bytes = 16384

A TCP frontend opts into SNI-scoped routing with hostname (mapped to the wire sni field) and, optionally, alpn:

[clusters.web-a]
protocol  = "tcp"
frontends = [
  { address = "0.0.0.0:8443", hostname = "a.example.com" },
]

[clusters.web-b]
protocol  = "tcp"
frontends = [
  # Only matches when the client also offers "h2" in its ALPN extension.
  { address = "0.0.0.0:8443", hostname = "b.example.com", alpn = ["h2"] },
]

[clusters.web-b-fallback]
protocol  = "tcp"
frontends = [
  # Catch-all for b.example.com when the client's ALPN offer doesn't match
  # any explicit entry above (or offers no ALPN at all). At most one
  # catch-all (empty `alpn`) per (address, hostname) is allowed.
  { address = "0.0.0.0:8443", hostname = "b.example.com" },
]

[clusters.legacy-passthrough]
protocol  = "tcp"
frontends = [
  # No `hostname` at all: the legacy single-cluster catch-all. A listener
  # cannot mix this shape with any SNI-scoped frontend above.
  { address = "0.0.0.0:9443" },
]

SNI matching

hostname accepts either an exact host ("example.com") or a single leading *. wildcard label ("*.example.com", matching a.example.com but neither the apex example.com nor a.b.example.com) — the same trie the HTTP/HTTPS router uses (crate::router::pattern_trie). No other wildcard shape is accepted (*.*.example.com, an embedded * outside the leading label, and a bare * are all rejected at config-load). The pattern must be plain ASCII: on-wire SNI is always an ASCII A-label (RFC 6066 §3 / IDNA), so write the punycode A-label form for non-ASCII hostnames (e.g. "xn--mnchen-3ya.example" for "münchen.example") rather than a Unicode U-label, which would load but never match.

ALPN matching

Within one (address, hostname) pair, route entries are tried in client preference order: the client's first offered protocol that is claimed by ANY entry's alpn list wins, regardless of route-table order. Only once no offered protocol matched any explicit entry does the catch-all (the one entry, if any, with an empty/absent alpn) win — including for a client that offers no ALPN extension at all.

Reject policy on an SNI-enabled listener

Once at least one SNI-scoped frontend targets a listener, EVERY connection on that listener is preread — including ones destined for a no-SNI-configured host. A connection is rejected (TCP FIN, no bytes forwarded) when:

• the first bytes aren't a TLS ClientHello at all (plain-TCP client, health check payload, ...);
• the ClientHello is malformed, or fragments past sni_preread_timeout without becoming complete;
• the ClientHello fails to complete within the first sni_preread_max_bytes bytes (the preread never reads past the cap, so a hello genuinely larger than it is rejected; a hello that completes within the cap routes no matter how much data the client has already coalesced after it — the excess stays in the kernel socket buffer and reaches the backend intact);
• the ClientHello carries no usable SNI (absent, empty, or hidden behind Encrypted Client Hello with no outer name);
• the SNI matches no configured hostname on this listener, or matches one but no route entry's alpn accepts the client's offer.

Each of these eleven rejection reasons has its own metric — see TCP SNI preread below. A bare TCP health check (connect + close, zero bytes) is not counted as a rejection — see that section's note.

Validation matrix (TOML config-load)

Configuration	Outcome
hostname is neither an exact host nor a single leading *. label	Error InvalidSniPattern
hostname contains a non-ASCII character	Error NonAsciiSniPattern — write the punycode A-label form
alpn is non-empty but hostname is absent	Error AlpnWithoutSni — a no-SNI frontend installs the raw catch-all path, which never consults alpn; the protocol list would silently never be enforced
Two frontends on the same (address, hostname) share an alpn protocol	Error TcpFrontendAlpnOverlap — routing must be deterministic, not iteration-order-dependent
More than one frontend on the same (address, hostname) omits alpn	Error TcpFrontendMultipleAlpnCatchAll — at most one catch-all per (address, hostname)
A listener has both a no-hostname frontend and at least one hostname-scoped frontend	Error TcpListenerMixesSniAndNoSni
sni_preread_timeout exceeds front_timeout on an SNI-enabled listener	Error SniPrereadTimeoutExceedsFrontTimeout — the preread phase cannot outlive the timeout that would already have closed the connection
sni_preread_max_bytes is below 5 bytes (a full TLS record header) on an SNI-enabled listener	Error SniPrereadMaxBytesTooSmall — 0 in particular makes the shell issue reads that can never make progress, spinning until the event-loop iteration guard trips
sni_preread_max_bytes exceeds the global buffer_size on an SNI-enabled listener	Error SniPrereadMaxBytesExceedsBufferSize — the preread buffer is carved out of the same per-session buffer used for relaying

**The routing-SHAPE validations — the mixing ban (TcpListenerMixesSniAndNoSni), ALPN-overlap / catch-all uniqueness (TcpFrontendAlpnOverlap / TcpFrontendMultipleAlpnCatchAll), ALPN-without-SNI (AlpnWithoutSni), and the pattern-SHAPE checks (InvalidSniPattern / NonAsciiSniPattern — is hostname an exact host or one leading *. label, every label non-empty, and pure ASCII) — are all enforced BOTH at TOML config-load (command/src/config.rs) AND on the worker's hot path (TcpListener::validate_new_tcp_front in lib/src/tcp.rs, which calls the identical shared pattern validator config-load uses), so an AddTcpFrontend sent directly over the command socket, a hot-added sozu frontend tcp add --sni, or a replay from a hand-edited/stale LoadState snapshot, is rejected exactly like a bad TOML file would be — a malformed pattern is never silently installed. Only the listener-level timeout/buffer checks (SniPrereadTimeoutExceedsFrontTimeout / SniPrereadMaxBytesTooSmall / SniPrereadMaxBytesExceedsBufferSize) still run at TOML config-load only: sozu listener tcp add/update accepts out-of-range preread knobs without error. An out-of-range sni_preread_max_bytes is at least degraded safely at the point of use — the worker clamps the effective cap to the session buffer's capacity and floors it at the 5-byte TLS record header, so a hot-added 0 cannot spin the preread loop — but a sni_preread_timeout above front_timeout is applied as-is during the preread phase. Prefer declaring TCP SNI routing in the TOML file and reloading, or double-check hot-added listener knobs carefully, until that remaining gap is closed.

Runtime CLI surface

sozu frontend tcp add \
  --id web-b \
  --address 0.0.0.0:8443 \
  --sni b.example.com \
  --alpn h2

sozu frontend tcp remove \
  --id web-b \
  --address 0.0.0.0:8443 \
  --sni b.example.com \
  --alpn h2

sozu listener tcp add \
  --address 0.0.0.0:8443 \
  --sni-preread-timeout 5 \
  --sni-preread-max-bytes 16384

--sni/--alpn are optional on both add and remove — omit both for a legacy no-SNI catch-all frontend. On remove, they must match the exact values the frontend was added with (same string, same wildcard shape, same alpn set): a mismatch removes nothing, silently, rather than erroring.

--sni-preread-timeout/--sni-preread-max-bytes are add-only: the UpdateTcpListenerConfig hot-reconfig message has no fields for them, so sozu listener tcp update cannot patch either knob on a live listener — remove and re-add the listener (or edit the TOML and reload) to change them.

Options specific to UDP listeners

A protocol = "udp" listener load-balances datagram traffic (DNS, syslog, NTP, generic UDP) to a cluster's backends. UDP is plaintext only and is served by exactly one worker per listener (see Limitations below).

[[listeners]]
protocol       = "udp"
address        = "0.0.0.0:53"
# public_address = "203.0.113.10:53"

# client / upstream flow idle timeout, in seconds. A flow is reaped once it has
# been idle for this long. Defaults to 30.
front_timeout  = 30
back_timeout   = 30

# maximum received datagram size, in bytes. Defaults to 1500 (one Ethernet
# frame). Capped at the global `buffer_size` (>= 16393): a value larger than
# `buffer_size` is clamped to it at config-load with a warning. A datagram on
# the wire that exceeds the effective read size is truncated (MSG_TRUNC) and
# dropped (`udp.datagrams.dropped`, reason `truncated`) — never a panic. Raise
# it for large EDNS0 / DNS responses; keep responses within PMTU where possible.
max_rx_datagram_size = 1500

# maximum number of concurrent flows on this listener. `0` (the default)
# auto-derives a cap from ~70% of the soft RLIMIT_NOFILE, because every flow
# owns one connected upstream socket (one fd). An explicit value above that
# bound is accepted but warns at config-load. New flows beyond the cap are shed
# (dropped + `udp.flows.shed`), never crashing the worker on EMFILE/ENFILE.
max_flows      = 0

Note: inbound UDP PROXY-protocol decode (expect_proxy) is not supported on UDP listeners — the field is rejected. (Sōzu can still send a PROXY v2 header to UDP backends; see the cluster send_proxy_protocol knob below.) UDP listeners also have no connect_timeout (there is no connect handshake) and no request_timeout.

UDP limitations

• One worker per UDP listener. A UDP listener is bound and served by exactly one worker, so per-listener throughput is one core. Scale across cores by running multiple UDP listeners (each owned by one worker) rather than expecting a single listener to span workers.
• Flows reset on hot-upgrade. On a zero-downtime upgrade the listener socket is handed to the new worker over SCM_RIGHTS, but per-flow state (the flow table and the per-flow connected upstream sockets) lives in the old worker's heap and is not migrated. The old worker drains its in-flight flows (bounded by the idle timeout) while the new worker serves new flows. For short datagram flows (DNS = one request + reply) this is near-invisible; a flow caught in-flight loses at most a datagram and the client retries.
• Plaintext only. No DTLS termination, no QUIC/HTTP3 CID-aware routing, no UDP-over-HTTP tunnelling.
• Other non-goals: Direct Server Return (DSR), io_uring / XDP / eBPF datapaths, eBPF SK_REUSEPORT single-listener multi-core, inbound UDP PROXY-protocol decode, and transparent (IP_TRANSPARENT) return are all out of scope.

Clusters

You can declare the list of your clusters under the [clusters] section. They follow the format:

Mandatory parameters:

[clusters]

[clusters.NameOfYourCluster]
# possible values are http or tcp
# https proxies will use http here
protocol = "http"

# per cluster load balancing algorithm. The possible values are
# "ROUND_ROBIN", "RANDOM", "LEAST_LOADED", "POWER_OF_TWO", "HRW" and
# "MAGLEV". HRW and MAGLEV are flow-affine policies designed for UDP
# clusters (see "UDP clusters" below). Defaults to "ROUND_ROBIN"
# load_balancing = "ROUND_ROBIN"

# force cluster to redirect http traffic to https
# https_redirect = true

frontends = [
  { address = "0.0.0.0:8080", hostname = "lolcatho.st" },
  { address = "0.0.0.0:8443", hostname = "lolcatho.st", certificate = "../lib/assets/certificate.pem", key = "../lib/assets/key.pem", certificate_chain = "../lib/assets/certificate_chain.pem" }
]
# additional options for frontends: sticky_session (boolean)

backends  = [
  { address = "127.0.0.1:1026" }
]

HTTP/2 backend connections (h2c)

By default, Sōzu speaks HTTP/1.1 to backend servers. You can enable cleartext HTTP/2 (h2c) for backend connections on a per-cluster basis using the http2 option:

[clusters.MyH2Cluster]
protocol = "http"
http2 = true

frontends = [
  { address = "0.0.0.0:8443", hostname = "app.example.com", certificate = "cert.pem", key = "key.pem", certificate_chain = "chain.pem" }
]
backends = [
  { address = "127.0.0.1:8080" }
]

When http2 = true, Sōzu opens cleartext HTTP/2 connections to the backend servers. This is useful when your backends natively support HTTP/2 (e.g., gRPC servers).

Important: http2 is a backend-capability hint — it tells Sōzu whether the backend speaks H2, nothing more. It does not gate H2 acceptance at the frontend. Frontend H2 is negotiated entirely via TLS ALPN (the alpn_protocols listener option) and is independent of per-cluster configuration. A cluster with http2 = false (or omitted) can still receive H2 requests from clients; Sōzu will translate them to H1 before forwarding to the backend. See command/src/config.rs:998 for the field definition.

You can also toggle HTTP/2 at runtime on an existing cluster via the CLI:

sozu cluster h2 enable --id MyH2Cluster
sozu cluster h2 disable --id MyH2Cluster

The frontend and backend protocols are independent. All four combinations work:

Client → Sōzu	Sōzu → Backend	Configuration
HTTP/1.1	HTTP/1.1	Default (no http2 flag)
HTTP/2	HTTP/1.1	Client negotiates H2 via ALPN, default backend
HTTP/1.1	HTTP/2	http2 = true on cluster
HTTP/2	HTTP/2	Client negotiates H2 via ALPN + http2 = true

Note: The http2 option controls the backend protocol only. The frontend protocol is determined by TLS ALPN negotiation between the client and Sōzu.

UDP clusters

A cluster fronting a UDP listener selects its load-balancing algorithm with the shared cluster-level load_balancing key, and carries its datagram-specific knobs (flow affinity, teardown counters, PROXY-protocol, health checks) under an optional [clusters.<id>.udp] block.

Two source-hash algorithms are added for flow-affine UDP selection (both are also valid for the existing load_balancing field):

load_balancing	Affinity	Notes
ROUND_ROBIN	none	Rotate through backends. The existing default; no source affinity.
HRW	yes	Highest-Random-Weight / rendezvous hashing — the recommended UDP default. O(N) per selection, no precomputed table, so no rebuild stall on hot-reconfig and provably-minimal flow remapping when backends change.
MAGLEV	yes	Maglev consistent hashing — O(1) per-packet table lookup, near-perfect balance, for large backend sets / high pps. The lookup table is rebuilt in the control-plane reconcile step on a backend-set change (not per-packet) and remaps ~2× more keys than HRW on churn.

Note: the load_balancing enum is parsed in SCREAMING_SNAKE_CASE ("ROUND_ROBIN", "HRW", "MAGLEV", …) — the same casing as every other proto-backed enum in this file (e.g. tls_versions).

The [clusters.<id>.udp] block:

Key	Default	Description
affinity_key	SOURCE_IP	Flow affinity key for hash LBs. SOURCE_IP pins every port from one client to one backend; SOURCE_IP_PORT keys on the full source 2-tuple.
responses	0	Expected replies per flow. A DNS flow sets responses = 1 so the flow closes immediately after its single reply; 0 = unlimited (syslog-style fire-and-forget).
requests	0	Maximum client datagrams per flow before teardown. 0 = unlimited.
send_proxy_protocol	false	Prepend a PROXY protocol v2 header (carrying the real client SocketAddr) to the backend. By default it is sent on the first datagram of the flow only. Backend PPv2-over-UDP parse support is not guaranteed by the spec — verify per backend.
proxy_protocol_every_datagram	false	When true, prepend the PPv2 header to every datagram instead of the first only. Useful when a flow may be re-created after idle eviction and the backend needs the client context on each datagram.

The [clusters.<id>.udp.health] sub-block configures active backend health checks, bound to the endpoint:

Key	Default	Description
mode	TCP_PROBE	HEALTH_OFF disables health checking; TCP_PROBE opens a non-blocking TCP connection to a companion port (the industry-standard liveness hint — a hint, not proof of UDP reachability); UDP_PROBE sends an application datagram and expects any reply. Defaults to TCP_PROBE when a […udp.health] block is present.
tcp_port	data port	Companion TCP probe port. Unset = the backend's data port.
rise	2	Consecutive successes before a backend is marked up (hysteresis).
fall	3	Consecutive failures before a backend is marked down (hysteresis). A flapping probe never re-hashes live flows; flows on a now-unhealthy backend stay pinned via the flow table until idle-timeout.
fail_open	true	When all backends read unhealthy, still run the normal LB over the full configured backend set (ignoring health) rather than black-holing traffic.
udp_probe_payload	—	Literal payload sent for a UDP_PROBE (e.g. a DNS query).
probe_interval_seconds	5	Delay between probes, in seconds.
probe_timeout_seconds	2	Per-probe response timeout, in seconds.

A complete DNS cluster, with HRW affinity, single-reply flows, PROXY v2 to the backend, and a TCP-probe health check:

[[listeners]]
protocol = "udp"
address  = "0.0.0.0:53"

[clusters.dns]
protocol       = "tcp"            # cluster transport family; UDP knobs live under [clusters.dns.udp]
load_balancing = "HRW"            # ROUND_ROBIN | HRW (source-hash, recommended for UDP) | MAGLEV
frontends = [
  { address = "0.0.0.0:53" }
]
backends = [
  { address = "10.0.0.10:53" },
  { address = "10.0.0.11:53" }
]

[clusters.dns.udp]
affinity_key        = "SOURCE_IP" # SOURCE_IP | SOURCE_IP_PORT
responses           = 1           # DNS = one reply per query; closes the flow when reached
requests            = 0           # 0 = unlimited client datagrams per flow
send_proxy_protocol = true        # PROXY v2 to the backend, first datagram only

[clusters.dns.udp.health]
mode      = "TCP_PROBE"           # HEALTH_OFF | TCP_PROBE | UDP_PROBE
tcp_port  = 53                    # companion probe port; default = the data port
rise      = 2
fall      = 3
fail_open = true                  # all-unhealthy ⇒ LB over the full configured set

Buffer size for HTTP/2

HTTP/2 uses a default maximum frame size of 16384 bytes (16 KiB) per RFC 9113 §6.5.2. Sōzu needs at least 9 additional bytes for the frame header (§4.1). Set buffer_size in the global section to at least 16393:

buffer_size = 16393

This is also the default — no action is needed unless an operator explicitly lowers buffer_size. As of PR #1209 Sōzu rejects start-up if buffer_size < 16393 and any HTTPS listener advertises h2 in its ALPN list, with a BufferSizeTooSmallForH2 error pointing at the conflicting listeners. The previous behaviour (silently accept the typo, then deadlock H2 mux on full-size frames) is gone.

To run with a smaller buffer, remove h2 from the listeners' alpn_protocols:

[[listeners]]
address = "0.0.0.0:443"
protocol = "https"
alpn_protocols = ["http/1.1"]   # h2 removed; buffer_size < 16393 is now valid

H2 flood detection thresholds

Sozu includes built-in flood detection for HTTP/2 connections. When a client sends an excessive number of certain frame types within a rolling window, Sozu terminates the connection with a GOAWAY(ENHANCE_YOUR_CALM) frame. This protects against several known HTTP/2 denial-of-service vectors.

Six per-window thresholds are configurable per-listener. When omitted, compile-time defaults are used (see also RST_STREAM lifetime caps for connection-lifetime counters):

Parameter	Default	Protects against	CVE
h2_max_rst_stream_per_window	100	Rapid Reset attack: client opens and immediately resets streams in a tight loop	CVE-2023-44487
h2_max_ping_per_window	100	Ping flood: client sends PING frames faster than the server can respond	CVE-2019-9512
h2_max_settings_per_window	50	Settings flood: client sends SETTINGS frames requiring ACKs, exhausting server resources	CVE-2019-9515
h2_max_empty_data_per_window	100	Empty DATA flood: client sends zero-length DATA frames to consume processing time	CVE-2019-9518
h2_max_window_update_stream0_per_window	100	Connection-level (stream 0) WINDOW_UPDATE flood: client sends a torrent of non-zero stream-0 WINDOW_UPDATE frames to burn server CPU parsing each one (zero-increment frames short-circuit into GOAWAY(PROTOCOL_ERROR) per RFC 9113 §6.9).
h2_max_continuation_frames	20	CONTINUATION flood: client sends many small CONTINUATION frames to exhaust header memory	CVE-2024-27316
h2_max_glitch_count	100	Cumulative protocol violations: total number of minor protocol errors before disconnection

Configuration example:

[[listeners]]
address = "0.0.0.0:443"
protocol = "https"

# H2 flood detection thresholds (optional, defaults shown)
h2_max_rst_stream_per_window = 100    # Rapid Reset (CVE-2023-44487)
h2_max_ping_per_window = 100          # Ping flood (CVE-2019-9512)
h2_max_settings_per_window = 50       # Settings flood (CVE-2019-9515)
h2_max_empty_data_per_window = 100    # Empty DATA flood (CVE-2019-9518)
h2_max_window_update_stream0_per_window = 100  # Connection-level WINDOW_UPDATE flood (stream 0)
h2_max_continuation_frames = 20       # CONTINUATION flood (CVE-2024-27316)
h2_max_glitch_count = 100             # Cumulative protocol violations

Note: When any threshold is exceeded, the connection is terminated with a GOAWAY frame using the ENHANCE_YOUR_CALM error code (HTTP/2 error code 0xb). The event is logged at warn level with the specific flood type that triggered disconnection.

Tuning h2_max_glitch_count in production

h2_max_glitch_count is a catch-all counter for low-severity protocol drift that no other flood counter covers. It is incremented on stream-close races (RST_STREAM / WINDOW_UPDATE / DATA on a closed stream), WINDOW_UPDATE with zero increment on a closed stream, and unknown SETTINGS identifiers. The counter uses a 1-second sliding window with half-decay (it halves at each window roll rather than resetting), so a threshold of N tolerates a one-shot burst of N glitches or a sustained rate of roughly N/2 glitches per second.

The default of 100 is conservative and protects a lightly-loaded edge well, but busy proxies that terminate aggressive-cancellation traffic (mobile clients, gRPC with deadlines, browser prefetch, fuzz harnesses) routinely trip it on legitimate races. If h2.flood.violation.glitch_window fires on traffic you know is benign, raise the threshold per-listener:

Traffic profile	Suggested h2_max_glitch_count
Default / low traffic	100
Busy public edge, mixed clients	500
gRPC / mobile / high cancellation	1000 – 2000
Load-test absorption only	5000

Before raising blindly, drop the relevant module to debug level and check which branch dominates — a single misbehaving backend or client emitting WINDOW_UPDATE on closed streams can be fixed upstream instead of hiding behind a larger budget. Never set the threshold to u32::MAX; the catch-all is the last line of defence against peers that stay just under every specific per-frame cap.

H2 connection tuning

Additional H2 parameters control connection-level behavior. All are optional per-listener with safe compile-time defaults:

Parameter	Default	Description
h2_initial_connection_window	1048576 (1MB)	Connection-level receive window size in bytes (RFC 9113 §6.9.2). Clamped to [65535, 2^31-1].
h2_max_concurrent_streams	100	Maximum concurrent H2 streams the proxy accepts (SETTINGS_MAX_CONCURRENT_STREAMS). Minimum: 1.
h2_stream_shrink_ratio	2	Shrink threshold ratio for recycled stream slots. The internal stream Vec is shrunk when total_slots > active_streams * ratio. Minimum: 2.
h2_max_header_list_size	65536	Maximum accumulated HPACK-decoded header list size per request (SETTINGS_MAX_HEADER_LIST_SIZE, RFC 9113 §6.5.2). Accounted as name + value + 32 octets per field, per §6.5.2; the per-field overhead also bounds the field count.
h2_stream_idle_timeout_seconds	max(30, back_timeout)	Per-stream idle timeout in seconds. An open H2 stream that receives no meaningful application data (non-empty DATA or HEADERS) for this duration is cancelled (RST_STREAM / CANCEL) to defend against slow-multiplex Slowloris. When unset the listener inherits back_timeout (floored at 30 s) so streams are not cancelled before the backend socket budget elapses; set explicitly to cap the per-stream deadline below back_timeout when under a slow-multiplex attack. Active uploads that trickle DATA frames reset the timer on each frame. The same deadline also governs a second, independent guard: a stream whose buffered response cannot drain because the peer keeps its receive window exhausted (HTTP/2 window-stall / WINDOW_UPDATE-drip) is reaped too. That flow-control-stall guard is NOT refreshed by inbound activity, so neither an inbound DATA drip nor a WINDOW_UPDATE(+1) drip can keep a window-stalled stream alive: the deadline clears only on a genuinely open send window or once cumulative outbound progress reaches one max DATA frame (16 KiB), so a trickle below that floor ages out and is reaped, while legitimate slow-but-steady transfers (sustaining more than ~0.5 KiB/s at the 30 s default) are unaffected. The guard is bidirectional — it also reaps a stalled request upload to a slow H2 backend (returning the client a 502), so raise this timeout if you proxy large uploads to slow H2 backends. A continuously window-blocked peer sustaining less than ~0.5 KiB/s for a full timeout is the one disclosed residual (raise the timeout to tolerate it).
h2_max_header_table_size	65536	Maximum HPACK dynamic table size (SETTINGS_HEADER_TABLE_SIZE) accepted from the peer. Caps the peer-advertised value to prevent unbounded HPACK encoder memory growth.
h2_max_header_fields	128	Maximum number of materialized header fields, enforced per HEADERS block and (independently) per trailers block — HPACK fields plus expanded cookie crumbs (RFC 9113 §8.2.3). Bounds the HPACK indexed-reference "header bomb", where 1-byte indexed references amplify into per-entry bookkeeping; cookie crumbs are counted individually (cf. Apache CVE-2026-49975). Minimum: 1.
h2_graceful_shutdown_deadline_seconds	5	Maximum wall-clock seconds to wait for in-flight H2 streams after GOAWAY(NO_ERROR) has been sent during soft-stop. Once the deadline elapses the connection is forcibly closed. Set to 0 to disable the forced close entirely — shutdown then waits for every stream to drain naturally (use with caution: a long-running request can delay the whole soft-stop indefinitely).

Configuration example:

[[listeners]]
address = "0.0.0.0:443"
protocol = "https"

# H2 connection tuning (optional, defaults shown)
h2_initial_connection_window = 1048576            # 1MB, min 65535, max 2147483647
h2_max_concurrent_streams = 100                   # min 1
h2_stream_shrink_ratio = 2                        # min 2
h2_max_header_list_size = 65536                   # HPACK decoded header budget
h2_stream_idle_timeout_seconds = 30               # per-stream idle timeout (default: max(30, back_timeout))
h2_max_header_table_size = 65536                  # HPACK dynamic table size cap
h2_max_header_fields = 128                         # max header fields/request (incl. cookie crumbs)
h2_graceful_shutdown_deadline_seconds = 5         # soft-stop forced-close deadline (0 = wait forever)

H2 RST_STREAM lifetime caps

In addition to the per-window h2_max_rst_stream_per_window threshold, three lifetime counters limit the total number of RST_STREAM frames associated with a single connection — two on the received side (Rapid Reset, CVE-2023-44487) and one on the emitted side (MadeYouReset, CVE-2025-8671). Together they catch patient-attacker patterns that stay just below the per-window threshold.

Parameter	Default	Description
h2_max_rst_stream_lifetime	10000	Absolute lifetime cap on RST_STREAM frames received on this connection.
h2_max_rst_stream_abusive_lifetime	50	Lifetime cap on "abusive" received RST_STREAM frames — resets sent by the peer before a response starts, the Rapid Reset signature (CVE-2023-44487).
h2_max_rst_stream_emitted_lifetime	500	Absolute lifetime cap on RST_STREAM frames emitted by the server (CVE-2025-8671 "MadeYouReset"). Increments on every non-NoError reset triggered by an attacker-crafted frame (Content-Length mismatch, header parse error, rejected priority, zero-increment WINDOW_UPDATE on an open stream). Graceful NoError cancels (stream recycle, propagated client cancel) are exempt. Crossing the threshold emits GOAWAY(EnhanceYourCalm).

Configuration example:

[[listeners]]
address = "0.0.0.0:443"
protocol = "https"

h2_max_rst_stream_lifetime = 10000
h2_max_rst_stream_abusive_lifetime = 50
h2_max_rst_stream_emitted_lifetime = 500

Security and protocol settings

Parameter	Default	Description
strict_sni_binding	true	Every HTTP request must have its :authority / Host covered by a SAN dNSName entry of the certificate served on this TLS session, with RFC 6125 §6.4.3 wildcard handling (CWE-346 / CWE-444). Per RFC 6125 §6.4.4, when the SAN dNSName list is present the Common Name is ignored; CN is only consulted as a fallback identity when the cert omits the SAN extension. Applies to HTTPS listeners only; plaintext listeners never have an SNI to compare against. Default-cert handshakes (rustls fell back to the bundled cert because no SAN matched the SNI) fall back to the legacy SNI exact-match check — pre-fix posture. Misses are answered with 421 Misdirected Request (RFC 9110 §15.5.20). This matches Firefox / Chrome HTTP/2 connection-coalescing semantics (RFC 7540 §9.1.1 / RFC 9113 §9.1.1) — browsers reuse one H2 connection for any origin covered by the served cert.
disable_http11	false	Only accept HTTP/2 connections; clients that do not negotiate h2 via TLS ALPN (including those that omit ALPN entirely) are dropped at handshake instead of silently downgrading to HTTP/1.1.

Configuration example:

[[listeners]]
address = "0.0.0.0:443"
protocol = "https"

strict_sni_binding = true   # require :authority covered by served cert SANs (RFC 6125 wildcards, default)
disable_http11 = false      # allow HTTP/1.1 fallback (default)
sozu_id_header = "Sozu-Id"  # rename the per-request correlation header (default "Sozu-Id")

The sozu_id_header knob renames the correlation header Sozu injects on every request AND response. Each request gets a unique ULID whose value is written to both sides — operators can grep the same identifier across client logs, proxy access logs, and backend logs. Default is Sozu-Id; a common rebrand is X-Request-Trace or X-Edge-Id. The value must be a valid HTTP header name (token chars per RFC 9110 §5.1). Applies to both HTTP and HTTPS listeners.

X-Real-IP injection and anti-spoof elision

Two listener-scoped flags control how Sōzu handles the X-Real-IP header. They default off (current behaviour: client-supplied value passes through, no proxy injection) and are independently combinable:

Parameter	Default	Description
elide_x_real_ip	false	Strip any client-supplied X-Real-IP header from forwarded requests before they reach the backend. Anti-spoofing — without this, a client can claim any value and downstream apps that key on X-Real-IP trust it verbatim.
send_x_real_ip	false	Append a proxy-generated X-Real-IP header carrying session_address.ip() — the connection peer IP after PROXY-v2 unwrap, i.e. the original client IP.

The four valid combinations cover the typical use cases:

elide_x_real_ip	send_x_real_ip	Behaviour
false	false	Default. Client header passes through, no proxy injection.
false	true	Send-only. Both client and proxy headers reach the backend. Operator caveat: backends that read the first X-Real-IP header (rather than the last) trust the client value over the proxy value. Use elide_x_real_ip = true alongside for full anti-spoofing.
true	false	Anti-spoof only. Client header stripped, no proxy injection.
true	true	Anti-spoof + send. Client header stripped, proxy header carries the connection peer IP.

The injected value is the IP of the connection peer after PROXY-v2 unwrap. When the listener has expect_proxy = true and the upstream sends a PROXY-v2 frame, session_address is rewritten to the original client IP before this header is generated, so the carried address is the real client even with one (or more) PROXY-v2 hops in front of Sōzu.

Both flags apply uniformly to HTTP/1 and HTTP/2 because the elision and injection live on the shared HttpContext::on_request_headers callback. The H2 trailer-block code path additionally honours elide_x_real_ip for trailer HEADERS frames, so an H2 client cannot spoof x-real-ip as a trailer to bypass the anti-spoof.

Both knobs are runtime-patchable via UpdateHttpListenerConfig / UpdateHttpsListenerConfig. Patches apply immediately to all H1 sessions and to new H2 connections; already-open H2 connections continue using the values captured at their handshake (same connection-scoped-capture semantic as strict_sni_binding). Long-lived H2 connections (CDN-style, mobile keep-alive) therefore observe a delayed flag flip — this is the established mux precedent, not a new defect.

[[listeners]]
address = "0.0.0.0:443"
protocol = "https"

elide_x_real_ip = true   # strip client-supplied X-Real-IP before forwarding
send_x_real_ip  = true   # inject a proxy-generated X-Real-IP carrying the peer IP

TLS handshake log severity cheat-sheet

Sōzu tiers the log severity of every rustls handshake error by root cause so scanner noise does not crowd out real configuration errors (commit 156cc217). Operators monitoring logs should expect the following classification:

Error variant	Level	Typical cause
NoApplicationProtocol, NoKxGroupsInCommon, NoCipherSuitesInCommon, NoCertificatesPresented	warn	Remote scanner / ssllabs probe with restricted cipher or ALPN set. Benign — no action required unless the client is legitimate.
InappropriateMessage, InappropriateHandshakeMessage, InvalidMessage, PeerMisbehaved, PeerIncompatible, PeerSentOversizedRecord	warn	Peer protocol violation — typically a broken or outdated TLS client, occasionally a port-scan. Review if persistent from a known client.
InvalidCertificate, DecryptError	warn	Client certificate rejected (e.g. expired, unknown CA) or bad shared key material. Relevant only when mTLS is configured.
AlertReceived(_)	debug	Client sent a TLS alert (e.g. user_canceled, close_notify_required) — conversation-level noise.
Everything else (server-side faults, unexpected internal errors)	error	Real bug or misconfiguration on the proxy. Actionable — triage immediately.

In particular, the recurring Could not look up a certificate for server name "<name>" line (issue #774) is emitted at warn when the rustls::server::ClientHello SNI does not match any configured frontend — the usual culprit is a generic scanner (Shodan, ssllabs, Censys) probing default.example or an IP-only hello. No action required unless the name matches a frontend you expect to serve.

Frontend matching: pairing, precedence, and patterns

This section covers operator-facing rules that surfaced as #1220, #1221, and #1222.

HTTPS listener pairing — frontends and cluster protocol

A protocol = "https" listener terminates TLS at the listener boundary. After termination, the request is plaintext HTTP/1.1 or HTTP/2 (the listener decides via ALPN), and the routed cluster sees it as protocol = "http". Frontends bound to an HTTPS listener address therefore declare protocol = "http" on the cluster — there is no separate protocol = "https" cluster mode.

Working configuration (one HTTP listener, one HTTPS listener, both routing to the same backend pool):

[[listeners]]
protocol = "http"
address = "[::]:80"

[[listeners]]
protocol = "https"
address = "[::]:443"
# Certificates are loaded via the runtime API or [listeners.https.certificates]
# (see "Listeners" §Certificates and TLS).

[clusters.example]
protocol = "http"  # plaintext after TLS termination at the listener
frontends = [
  { address = "[::]:80",  hostname = "subdomain.domain.tld" },
  { address = "[::]:443", hostname = "subdomain.domain.tld" },
]
backends = [
  { address = "[::1]:8080" },
]

Sōzu refuses to start when an HTTPS listener has no matching frontend — no graceful fallback. The listener-end-to-end design is intentional: TLS settings (cipher list, ALPN, SNI binding, X-Real-IP elision) live on the listener; per-cluster TLS-to-backend remains a separate v2.1.0 feature (#1218).

Path matching precedence within a frontend

Path filtering in Sōzu is frontend-level, not backend-level. Backends inside a cluster cannot be restricted to a path; if you need different backend pools for / and /some-path on the same hostname, declare two clusters that share the hostname and differ on the frontend path rule. For example, sending example.com to one backend pool and example.com/instance to another:

[clusters.example-main]
protocol = "http"
frontends = [
  { address = "0.0.0.0:443", hostname = "example.com" },
]
backends = [
  { address = "127.0.0.1:8001" },
  { address = "127.0.0.1:8002" },
]

[clusters.example-instance]
protocol = "http"
frontends = [
  { address = "0.0.0.0:443", hostname = "example.com", path = "/instance" },
]
backends = [
  { address = "127.0.0.1:8011" },
  { address = "127.0.0.1:8012" },
]

A request to https://example.com/instance/foo matches the longer prefix /instance and lands on example-instance; everything else falls back to the empty-prefix frontend on example-main. Per-frontend certificate / key / certificate_chain fields are omitted here for brevity but are valid on each frontend entry.

When multiple frontends share the same (address, hostname) tuple and differ only on path / path_type, the lookup picks the most specific rule using this fixed precedence:

1. path_type = "EQUALS" — exact match wins first.
2. path_type = "REGEX" — anchored at both ends (\A...\z) since v2.0.0; longer literal substrings within the regex are not weighted, so multiple regex rules competing on the same authority produce undefined ordering between them.
3. path_type = "PREFIX" — fall-through default. Longest prefix wins among PREFIX rules.

Operators wanting to route /.well-known/acme-challenge separately from / on the same hostname should declare:

[[clusters.acme-helper.frontends]]
address = "[::]:80"
hostname = "mail.example.tld"
path = "/.well-known/acme-challenge"
path_type = "PREFIX"

[[clusters.mail-server.frontends]]
address = "[::]:80"
hostname = "mail.example.tld"
path = "/"
path_type = "PREFIX"

Both PREFIX rules match a request, but the longer prefix (/.well-known/acme-challenge) wins per the longest-match rule above. Configuration order does not affect routing: lookup is by trie specificity, not declaration order.

Regex limitations — no look-around

Sōzu uses the standard Rust regex crate which does not support look-around (negative or positive lookahead / lookbehind). Patterns like ^(?!\/\.well-known\/acme-challenge) will not compile or will be rejected at frontend registration.

If you need "match all paths except X", decompose into two PREFIX rules per the example above and rely on the longest-prefix rule, or declare an explicit list of EQUALS rules.

The fancy-regex crate supports look-around but is not currently a Sōzu dependency. #1222 tracks the discussion if look-around becomes a real operator constraint.

send_proxy and expect_proxy apply to TCP only

The PROXY-protocol forwarding/expecting flags send_proxy = true and expect_proxy = true on a backend or frontend are valid only for clusters with protocol = "tcp". HTTP/HTTPS clusters use the forwarding HTTP headers (X-Forwarded-For, X-Real-IP, and — once v2.1.0 lands — RFC 7239 Forwarded) instead.

Setting send_proxy = true on an HTTP/HTTPS cluster is silently ignored at runtime. This contradicts a literal reading of #1221; the documentation tracks reality but operators upgrading from older configs should remove the no-op flag from HTTP cluster definitions.

See #PROXY Protocol below for the supported TCP-mode shape.

Custom HTTP answer templates

Sōzu lets operators replace any default error response with a templated body. Templates are specified as a [listeners.<id>.answers] map at listener scope — the global default that fires whenever no cluster-level override matches — or as a [clusters.<id>.answers] map at cluster scope, which overrides the matching listener entry on requests routed through that cluster. Both layers accept the same key/value shape: the key is the HTTP status code (e.g. "503"); the value is either an inline literal body (the default — the value is taken verbatim) or file://<path> to load the body off disk. The inline form is convenient for short canned responses, secrets-free containers where mounting a template directory is awkward, and test rigs that want to avoid disk dependencies.

# listener-level: global default for every status not overridden by a cluster.
[listeners.https.answers]
"401" = "file:///etc/sozu/templates/401.http"        # load from disk
"404" = "file:///etc/sozu/templates/404.http"
"503" = """HTTP/1.1 503 Service Unavailable\r\nConnection: close\r\nContent-Length: 4\r\n\r\nbusy"""  # inline literal (no prefix)

# cluster-level: overrides the listener default for THIS cluster only.
[clusters.MyCluster.answers]
"503" = "file:///etc/sozu/templates/MyCluster.503.http"

Each template is a complete HTTP response (status line + headers + body) with optional placeholders. Sōzu substitutes the placeholders at render time:

Placeholder	Scope	Meaning
%REQUEST_ID	header & body	Per-request ULID (matches Sozu-Id)
%CLUSTER_ID	header & body	Cluster the request was routed to (or empty)
%BACKEND_ID	header & body	Backend the request was forwarded to (or empty)
%ROUTE	header & body	Request method + authority + path
%REDIRECT_LOCATION	header & body	Resolved Location URL (301 only)
%WWW_AUTHENTICATE	header only	Realm string for 401 (header is elided when empty)
%MESSAGE, %PHASE, %SUCCESSFULLY_PARSED, %PARTIALLY_PARSED, %INVALID, %CAPACITY, %DURATION	varies	Diagnostic detail for parse / size / timeout errors

When the template carries a Content-Length: <N> header, the engine recomputes the value from the actual rendered body size after %-substitutions — so a literal value that drifted from the body length cannot land on the wire (RFC 9110 §8.6 / RFC 7230 §3.3.2 anti- smuggling). Templates that omit Content-Length keep the byte-for-byte shape they were written with; nothing is synthesised. Operators who want a Content-Length include one; those who rely on Connection: close for body framing get a clean header-only response.

When a template carries Connection: close, the response will close the frontend connection after delivery; a custom template without that header keeps frontend keep-alive on. The HAProxy parallel is errorfile NNN /path.

The legacy answer_NNN = "/path" per-status fields under [listeners.<name>] continue to work — they are merged into the new map at load time so existing state files round-trip cleanly. New configs should prefer the [listeners.<name>.answers] shape.

Frontend redirect, URL rewrite, and custom headers

Each frontend can carry a routing decision richer than "forward to a cluster". Three top-level knobs drive different policies:

[[clusters.MyCluster.frontends]]
address  = "0.0.0.0:80"
hostname = "old.example.com"
# Force a permanent 301 to the canonical name on a different port.
redirect        = "permanent"      # forward (default) | permanent | found | permanent_redirect | unauthorized
redirect_scheme = "use-https"      # use-same (default) | use-http | use-https
rewrite_host    = "new.example.com"
rewrite_port    = 8443             # paired with `cluster.https_redirect_port`

redirect = "permanent" returns a 301 with a resolved Location URL built from redirect_scheme, the (optionally rewritten) host, the cluster's https_redirect_port (or the rewritten rewrite_port), and the original request path. redirect = "found" and redirect = "permanent_redirect" share the same Location-resolution machinery and emit 302 (RFC 9110 §15.4.3) or 308 (RFC 9110 §15.4.9) respectively. The semantic distinction matters to clients: 301 and 302 permit user agents to rewrite POST → GET on follow, whereas 308 (and the IETF-modern 307) MUST preserve the request method. Pick permanent_redirect (308) when you want a stable cacheable redirect that keeps method semantics — e.g. an API endpoint relocation. Pick found (302) for short-lived forwards (login flows, A/B-testing gates). redirect = "unauthorized" returns 401 unconditionally — useful for blanket deny-by-default frontends that still want to surface a login prompt with the cluster's www_authenticate realm.

rewrite_host and rewrite_path accept a small template grammar: $HOST[n] references the n-th host capture (0 is the full hostname; 1+ are regex / wildcard subgroups), and $PATH[n] references the n-th path capture. When the host is rewritten, Sōzu injects the original host into X-Forwarded-Host so backends can reconstruct the request URL.

Custom headers attach to the same frontend:

[[clusters.MyCluster.frontends.headers]]
position = "request"               # request | response | both
key      = "X-Forwarded-Proto"
value    = "https"

[[clusters.MyCluster.frontends.headers]]
position = "response"
key      = "X-Cache-Backend"
value    = ""                      # empty value DELETES the header by name
                                   # (HAProxy `del-header` parity)

HAProxy parallels: http-request redirect (permanent / scheme), set-uri and set-path (rewrite), http-request set-header and http-request del-header (custom headers).

HSTS for the HTTPS counterpart of an automatic redirect

When an HTTP listener carries a redirect_scheme = "use-https" frontend that pushes traffic to an HTTPS counterpart, HSTS belongs on the HTTPS side, never on the redirect itself. RFC 6797 §7.2 forbids Strict-Transport-Security on plaintext-HTTP responses; conformant browsers ignore the header on plain-HTTP and Sōzu refuses the configuration at load time on HTTP listeners.

The recommended pattern is two paired frontends with HSTS attached to the HTTPS one:

# Plain-HTTP frontend that pushes everything to HTTPS via 301.
[[clusters.api.frontends]]
address         = "0.0.0.0:80"
hostname        = "api.example.com"
redirect        = "permanent"     # 301 (or "permanent_redirect" for 308)
redirect_scheme = "use-https"
# NO [hsts] block here — Sōzu would reject it (RFC §7.2).

# HTTPS counterpart that serves traffic and pins HSTS.
[[clusters.api.frontends]]
address  = "0.0.0.0:443"
hostname = "api.example.com"

[clusters.api.frontends.hsts]
enabled            = true
max_age            = 31536000     # 1 year — preload list minimum
include_subdomains = true
preload            = false        # opt-in only

For convenience, declare HSTS once at the listener default and let every HTTPS frontend inherit:

[[listeners]]
protocol = "https"
address  = "0.0.0.0:443"

[hsts]
enabled            = true
max_age            = 31536000
include_subdomains = true

Per-frontend hsts.enabled = false suppresses the inherited default for a single frontend that cannot commit to HSTS yet.

The header is emitted on every successful HTTPS response, including proxy-generated 3xx redirects (e.g. redirect_scheme = "use-https" from an HTTPS frontend, or a redirect = "permanent" shape served on the HTTPS side), 401 auth-deny, and 502 / 503 / 504 default answers — RFC 6797 §8.1 requires HSTS on every response code from the host.

HTTP Basic authentication on a frontend

Operators can require a valid Authorization: Basic header on a frontend and validate it against a list of pre-hashed credentials on the cluster. The mux iterates the entire authorized list in constant time (via subtle::ConstantTimeEq), so neither the matching index nor a successful lookup leak through the time spent validating.

[clusters.MyCluster]
# Realm rendered into `WWW-Authenticate: Basic realm="…"` on a 401.
www_authenticate = 'Basic realm="MyCluster"'
# Each entry is `username:hex(sha256(password))`. The runtime hashes
# ONLY the password — the username appears verbatim before the colon
# and is NOT part of the hashed input. Generate the hash for password
# `secret` with:
#   printf 'secret' | sha256sum | awk '{print $1}'
# `printf` (not `echo`) omits the trailing newline so the digest matches
# the bytes carried in `Authorization: Basic <base64>`.
authorized_hashes = [
    "admin:2bb80d537b1da3e38bd30361aa855686bde0eacd7162fef6a25fe97bf527a25b",
]

[[clusters.MyCluster.frontends]]
address       = "0.0.0.0:80"
hostname      = "secured.example.com"
required_auth = true               # gate this frontend on basic-auth

Failure modes:

• Missing Authorization: Basic header → 401 with the cluster's realm
• Malformed credential (bad base64, missing :) → 401
• Wrong username or password → 401
• Empty authorized_hashes while the frontend has required_auth = true → 401 (closed-by-default policy)

The WWW-Authenticate header is rendered through the answer template's %WWW_AUTHENTICATE placeholder; when no realm is configured the entire header line is elided from the 401 response. HAProxy parallel: http-request auth realm Foo unless { http_auth(...) }.

Tuning the credential decode cap

A hostile peer can send arbitrarily long Authorization: Basic <token> values. Sōzu base64-decodes the token in a transient allocation; an unbounded decode per failed-auth attempt is a memory pressure vector. The worker caps the decoded length to 4096 bytes by default — well above the realistic username:password shape (typical credentials are <100 bytes). Operators running hardened tenants can lower this in the main TOML config:

# in the top-level config (alongside `buffer_size`, `max_connections`, etc.)
basic_auth_max_credential_bytes = 256

The override is committed once on each worker at boot and applies to every cluster. Setting 0 is a no-op (the built-in default stays in force) so an explicit zero in a config file does not accidentally disable the cap.

The config validator emits a warn! line at boot when the configured cap is >= buffer_size / 3: at that point a single failed-auth attempt can pin ~33% of the per-frontend buffer's worth of bytes, which combined with in-flight request/response framing pushes the buffer toward back-pressure under load. The warning is informational only — operators with a deliberate trade-off can keep the value, but the surprise stays visible in the boot log.

Runtime patch — sozu listener update

sozu listener {http,https,tcp} update patches a live listener in place without cycling the listening socket. Only the fields you pass are written; all others are preserved exactly as they are. Existing sessions continue with their configuration snapshot; only new sessions, connections, or TLS handshakes — depending on the field — pick up the new values. Use sozu listener list to inspect current values before patching.

Bind-only fields are not patchable. The address, TLS crypto parameters, and the active flag can only be changed by removing and re-adding the listener:

sozu listener https remove -a 0.0.0.0:8443
sozu listener https add --address 0.0.0.0:8443 [...]

Bind-only fields: address, tls_versions, cipher_list, cipher_suites, signature_algorithms, groups_list, certificate, certificate_chain, key, send_tls13_tickets, active.

Updatable fields

Fields are grouped by the earliest point at which a patched value takes effect for connections already in progress. All fields apply to new sessions immediately after the patch is acknowledged.

HTTP and HTTPS listeners

Field	Type	Mutability class	Default	Notes
public_address	SocketAddr	session-at-accept	—	Source address reported to backends / logs
expect_proxy	bool	session-at-accept	false	Enable PROXY protocol v1/v2 on new sessions
sticky_name	string	session-at-accept	"SOZUBALANCEID"	Sticky-session cookie name
front_timeout	u32 (seconds)	session-at-accept	60	Max idle time on the client socket
back_timeout	u32 (seconds)	session-at-accept	30	Max idle time on the backend socket
connect_timeout	u32 (seconds)	session-at-accept	3	Max time to establish a backend connection
request_timeout	u32 (seconds)	session-at-accept	10	Max time to send a complete request
http_answers	file paths	session-at-accept	built-in defaults	Listener-default HTTP error bodies (301/401/404/408/413/421/502/503/504/507). Per-cluster answer_503 overrides are preserved.
sozu_id_header	string	session-at-accept	"Sozu-Id"	Correlation header name (RFC 9110 §5.1 token; reject empty or containing CR/LF/colon/space)
h2_max_rst_stream_per_window	u32 (≥ 1)	per-connection setup	100	RST_STREAM flood cap — CVE-2023-44487, CVE-2019-9514
h2_max_ping_per_window	u32 (≥ 1)	per-connection setup	100	PING flood cap — CVE-2019-9512
h2_max_settings_per_window	u32 (≥ 1)	per-connection setup	50	SETTINGS flood cap — CVE-2019-9515
h2_max_empty_data_per_window	u32 (≥ 1)	per-connection setup	100	Empty DATA flood cap — CVE-2019-9518
h2_max_continuation_frames	u32 (≥ 1)	per-connection setup	20	CONTINUATION flood cap — CVE-2024-27316
h2_max_glitch_count	u32 (≥ 1)	per-connection setup	100	Cumulative protocol-anomaly budget
h2_max_window_update_stream0_per_window	u32 (≥ 1)	per-connection setup	100	Stream-0 WINDOW_UPDATE flood cap
h2_max_rst_stream_lifetime	u64 (≥ 1)	per-connection setup	10000	Lifetime RST_STREAM received cap — CVE-2023-44487
h2_max_rst_stream_abusive_lifetime	u64 (≥ 1)	per-connection setup	50	Lifetime abusive RST_STREAM cap (Rapid Reset signature)
h2_max_rst_stream_emitted_lifetime	u64 (≥ 1)	per-connection setup	500	Lifetime server-emitted RST_STREAM cap — CVE-2025-8671
h2_initial_connection_window	u32	per-connection setup	1048576	Connection receive window (bytes, RFC 9113 §6.9.2)
h2_max_concurrent_streams	u32 (≥ 1)	per-connection setup	100	SETTINGS_MAX_CONCURRENT_STREAMS
h2_stream_shrink_ratio	u32 (≥ 2)	per-connection setup	2	Stream-slot Vec shrink threshold
h2_max_header_list_size	u32	per-connection setup	65536	HPACK decoded header budget (SETTINGS_MAX_HEADER_LIST_SIZE)
h2_max_header_table_size	u32	per-connection setup	65536	HPACK dynamic table size cap (SETTINGS_HEADER_TABLE_SIZE)
h2_stream_idle_timeout_seconds	u32	per-connection setup	max(30, back_timeout)	Per-stream idle timeout (slow-multiplex Slowloris defence). When unset, inherits back_timeout floored at 30 s; set explicitly to cap below back_timeout.
h2_graceful_shutdown_deadline_seconds	u32	per-connection setup	5	Forced-close deadline after GOAWAY(NO_ERROR) on soft-stop. 0 = wait forever.

HTTPS-only fields

Field	Type	Mutability class	Default	Notes
alpn_protocols	string[]	per-handshake	["h2","http/1.1"]	Rebuilds the rustls ServerConfig. In-flight handshakes finish on the old config. Pass --reset-alpn on the CLI to restore the default.
strict_sni_binding	bool	per-handshake	true	Require :authority/Host covered by served cert SAN dNSName (RFC 6125 §6.4.3/6.4.4, CWE-346/CWE-444). Default-cert handshakes fall back to legacy SNI exact-match. Miss → 421 (RFC 9110 §15.5.20).
disable_http11	bool	per-handshake	false	Drop clients that do not negotiate h2 via ALPN

TCP listeners

Field	Type	Mutability class	Default	Notes
public_address	SocketAddr	session-at-accept	—
expect_proxy	bool	session-at-accept	false
front_timeout	u32 (seconds)	session-at-accept	60
back_timeout	u32 (seconds)	session-at-accept	30
connect_timeout	u32 (seconds)	session-at-accept	3

UDP listeners

Patched values apply to new flows; flows already in progress keep their captured value. address and active are bind-only — change them with RemoveListener + add.

Field	Type	Mutability class	Default	Notes
public_address	SocketAddr	flow-at-admit	—	Source address reported to backends / logs
front_timeout	u32 (seconds)	flow-at-admit	30	Client flow idle timeout
back_timeout	u32 (seconds)	flow-at-admit	30	Upstream flow idle timeout
max_rx_datagram_size	u32 (bytes)	flow-at-admit	1500	Clamped to the effective buffer_size
max_flows	u32	flow-at-admit	0	0 = auto (~70% of soft RLIMIT_NOFILE)

Examples

Tighten H2 flood thresholds under attack (CVE-2023-44487)

# Inspect current values first
sozu listener list

# Halve the Rapid Reset budget on the HTTPS listener
sozu listener https update -a 0.0.0.0:8443 \
    --h2-max-rst-stream-per-window 50 \
    --h2-max-rst-stream-abusive-lifetime 25

# Confirm the new values are live
sozu listener list

Existing H2 connections continue with their original thresholds. New connections opened after the patch acknowledge see the tighter limits.

Toggle disable_http11 to enforce H2-only mode

# Require all clients to negotiate h2 via TLS ALPN
sozu listener https update -a 0.0.0.0:8443 --disable-http11

# Revert — allow HTTP/1.1 fallback again
sozu listener https update -a 0.0.0.0:8443 --enable-http11

The existing HTTP/1.1 sessions in progress are not disrupted; only new TLS handshakes that omit h2 from their ALPN offer are refused after the patch.

Rebrand the correlation header

# Rename "Sozu-Id" to "X-Edge-Id" organisation-wide
sozu listener http  update -a 0.0.0.0:80   --sozu-id-header "X-Edge-Id"
sozu listener https update -a 0.0.0.0:8443 --sozu-id-header "X-Edge-Id"

The new header name takes effect for sessions accepted after the patch. Previously accepted sessions continue to inject the old Sozu-Id header until they close.

Observability

Two worker metrics track update outcomes:

Metric	Description
listener.updated	Incremented each time a worker successfully applies a patch
listener.update_failed	Incremented when the worker-side apply returns an error

The control-plane command server also emits a LISTENER_UPDATED event on the SubscribeEvents bus (carrying the listener address and type) and writes a structured audit log line at info! level in the MUX Session(...) layout. See observability.md for the full format.

The same bus carries EventKind::ClusterRecovered (proto tag 29) on the AllDown → Available transition for any cluster that previously emitted EventKind::NoAvailableBackends. Subscribers that already track the all-down event get the recovery side for free without polling cluster.available_backends.

Access-log message field on timeout

When a session terminates on a timeout, Sōzu populates the access-log message field with a stable, structured token so dashboards and log-pipeline rules can attribute the outcome without inspecting the HTTP status. The vocabulary is closed (operator-visible API once shipped):

Token	Trigger	Status seen by client
client_timeout	Frontend timer fired while waiting for the request to arrive (TimeoutStatus::Request in the H1 path; StreamState::Idle in the mux path)	408 Request Timeout
client_timeout_during_response	Frontend timer fired while the backend was still composing the response — ambiguous case where timeout responsibility should already have switched. Mapped to gateway-timeout for client clarity	504 Gateway Timeout
backend_timeout	Backend timer fired before any response byte arrived — connection-level slowness or backend stuck pre-headers. The H1 invariant-break arm (TimeoutStatus::Request on the backend) collapses into the same token because the operator-visible cause is identical; the internal error! log keeps the diagnostic signal for sozu maintainers	504 Gateway Timeout
backend_response_timeout	Backend timer fired while the response body was streaming — partial response in flight. Mux replies with RST_STREAM (H2Error::InternalError); H1 forcibly closes the session because no default-answer can replace an in-flight response body	RST_STREAM (mux) / connection close (H1)

Non-timeout default-answer paths (e.g. 503 from Router::route_from_request, 401 from auth, 301 from redirects) leave the field as None and the access log emits message: None exactly as before. Sessions that complete normally are also unaffected.

Backend health checks

You can optionally configure active HTTP health checks for backends. See health_checks.md for full details.

[clusters.NameOfYourCluster.health_check]
uri = "/health"
interval = 10
timeout = 5
healthy_threshold = 3
unhealthy_threshold = 3
expected_status = 0

Metrics

Sōzu reports its own state to another network component through a UDP socket. The main process and the workers are responsible to send their states. We implement the statsd protocol to send statistics. Any service that understands the statsd protocol can then gather metrics from Sōzu.

Architecture

Metrics are collected via thread-local storage macros (count!, gauge!, gauge_add!, time!, incr!, decr!) and dispatched to two drains:

• Local drain: Accumulates metrics in-memory with HDR histograms for latency percentiles. Queried via the CLI (sozu metrics get).
• Network drain: Sends metrics over UDP using the statsd protocol. Supports both plain dotted format and InfluxDB-style tagged format.

Metric types:

Type	Macro	StatsD suffix	Description
Counter	count!, incr!, decr!	|c	Monotonically increasing. Network drain: zeroed after each successful UDP send (per-interval delta). Local drain: cumulative since worker start.
Gauge	gauge!, gauge_add!	|g	Snapshot value (absolute or delta). Last-value-wins; never automatically reset.
Time	time!	|ms	Latency in milliseconds. Local drain stores an HDR histogram (Histogram<u64>, sigfig=3); samples accumulate since worker start.

Metrics have three scopes:

• Proxy-level: Global to the worker (no cluster or backend context)
• Cluster-level: Tagged with a cluster_id
• Backend-level: Tagged with both cluster_id and backend_id

Configure metrics

In your config.toml, you can define the address and port of your external service by adding:

[metrics]
address = "127.0.0.1:8125"
# use InfluxDB's statsd protocol flavor to add tags
# tagged_metrics = false
# metrics key prefix
# prefix = "sozu"
# cardinality knob — defaults to "cluster" (preserves historical behaviour)
# detail = "cluster"

Metrics are sent at most once per second per key (if updated). Cluster/backend metrics that have not been updated for 10 minutes are automatically dropped from the network drain.

Metrics scopes (local CLI vs network drain)

• The network drain (StatsD UDP) sends per-second deltas; counters are zeroed after each successful send. On RemoveCluster / RemoveBackend, the drain immediately drops the cluster's cluster_metrics, backend_metrics, and queued MetricLine entries — any unsent statsd interval for the cluster is discarded (no final flush). Without an explicit RemoveCluster, the existing 10-minute idle GC still applies.
• The local drain (queryable via sozu metrics) is cumulative since worker start for both proxy_metrics and per-cluster / per-backend cluster_metrics. There is no implicit hourly reset. Operators reset it explicitly with sozu metrics clear, which wipes everything (counts, gauges, histograms, proxy-wide and per-cluster, AND the master-process main_metrics aggregator).
• Per-cluster local-drain entries are dropped on RemoveCluster / RemoveBackend so the keyspace is bounded by the live configuration.
• Implication for dashboards: counters in sozu metrics output are monotonic. Charts must compute rate() / irate() rather than treat successive snapshots as windowed counts. Histograms accumulate every sample since worker start, so percentiles in the CLI snapshot are lifetime values, not windowed.

Cardinality knob (metrics.detail)

Mirrors HAProxy's extra-counters opt-in: operators choose the lowest level that satisfies their dashboards so the StatsD keyspace stays bounded. The level filters the (cluster_id, backend_id) labels at emission time — both the local CLI drain and the network drain see the same filtered labels, so dashboards stay consistent.

Each level is a SUPERSET of the previous one:

Level	Behaviour	Use when
process	Both cluster_id and backend_id are dropped — proxy-only counters.	The smallest possible keyspace; dashboards aggregate everything at the worker level.
frontend	Same as process today. Reserved for the per-listener (frontend) counters that are tracked as a follow-up — operators can opt in already, the value is just stored on ServerMetricsConfig.detail and applied on every emission.	Forward-compatible config that picks up per-listener counters when they ship without a config-file change.
cluster	Keeps cluster_id, drops backend_id. Default — preserves the historical pre-knob behaviour.	The current shape of every existing dashboard.
backend	Keeps both labels. Highest cardinality.	Per-backend SLOs / hotspot debugging when the cluster has few backends.

Memory note for Time histograms: per-bucket counters are Histogram<u64> (widened from Histogram<u32> to avoid saturation at sustained high-RPS — u32 saturates a single popular bucket in ≈72 minutes at 1 M samples/s in a 64-bit count). Memory footprint per Time histogram is bounded by hdrhistogram's sigfig=3 shape, roughly 16–32 KB per histogram, doubled vs. the prior u32 shape. With cluster-level cardinality this multiplies by the number of live clusters × Time-key distinct names; with backend-level, also by per-cluster backend count. Plan capacity accordingly.

Filtering happens centrally in Aggregator::receive_metric (lib/src/metrics/mod.rs) via the pure helper filter_labels_for_detail, unit-tested exhaustively across all four levels. Workers receive the level over the SCM socket as a proto enum (MetricDetail); old binaries on either side fall back to cluster so a mixed-version rollout keeps emitting the historical metric shape.

Runtime cardinality lease

Operators can elevate a worker's effective metric_detail for the lifetime of an interactive session without rewriting config.toml. sozu top uses this mechanism to enable backend detail while the TUI is attached, and reverts it on exit. Other tooling (per-host scraper agents, ad-hoc debugging) can use the same surface.

The lease is keyed by an operator-supplied client_id and stored on each worker. The effective level is max(configured, max(active leases)), so a lease never lowers the configured detail; it only elevates. When the last lease expires (TTL pop) or is explicitly cleared, the effective level falls back to the configured value.

The proto verb that exposes this surface is SetMetricDetail (request tag 55); the response shape is MetricDetailStatus carrying the master's (configured, effective, previous_effective) triple plus a per-worker WorkerMetricDetailStatus map. Every apply, clear, and TTL expiry emits an audit-log event of kind MetricDetailChanged (EventKind::METRIC_DETAIL_CHANGED, tag 30) on the text and JSON sinks, with lease_id= and metric_detail_reason= as dedicated columns so operator-supplied strings cannot smuggle a forged adjacent column.

Server-side caps and defaults (all defined in lib/src/metrics/mod.rs):

Knob	Default	Cap	Notes
ttl_seconds	60	300	Lease lifetime. The TUI renews every ttl/2 seconds; on TUI crash the lease self-expires after at most one ttl window. The master rejects out-of-range TTLs before fan-out so a buggy or malicious request cannot N×amplify worker-side rejections.
LEASE_TABLE_CAP	—	64	Maximum number of simultaneous leases per worker. Renewals of existing entries always succeed; only new inserts are subject to this cap. Mitigates the CWE-770 vector where a misbehaving client rolls client_id faster than expiry.
LEASE_CLIENT_ID_MAX_BYTES	—	64	Maximum client_id length. Operator tooling should pick a stable identifier (the TUI uses top:<pid>:<8-hex>); arbitrary user input must be capped at this length before submission.

Trust model: the lease records the connecting peer's PID and master session ULID (from SO_PEERCRED on Linux). Subsequent clear requests are authorised against the apply-time binding — a different operator on the same host cannot clear another operator's lease even if they guess the client_id. Pre-binding callers and platforms without SO_PEERCRED degrade to "binding unknown" → accept any clear. Clients NEVER set the peer fields themselves; the master populates them from the ClientSession before fan-out.

Reversibility: leases self-expire, so a crashed dashboard does not leave a worker permanently emitting at elevated cardinality. The audit-log trail records every transition; SOC tooling can reconstruct the cardinality posture of every worker at any point in time without polling.

Renewals are not state transitions — SetMetricDetail is deliberately not in the systemd RELOADING=1 / READY=1 bracket set, so a long-lived TUI session does not flap the unit state. The audit-log event remains the authoritative trail for cardinality changes.

StatsD wire format

Untagged (default, tagged_metrics = false):

sozu.WRK-00.http.requests:1|c
sozu.WRK-00.cluster.my-cluster.http.errors:0|c
sozu.WRK-00.cluster.my-cluster.backend.backend-1.backend_response_time:125|ms

Tagged (InfluxDB format, tagged_metrics = true):

sozu.http.requests,origin=WRK-00,version=1.1.0:1|c
sozu.cluster.http.errors,origin=WRK-00,version=1.1.0,cluster_id=my-cluster:0|c
sozu.backend.backend_response_time,origin=WRK-00,version=1.1.0,cluster_id=my-cluster,backend_id=backend-1:125|ms

Available metrics

Sōzu emits the following metrics via statsd. All metrics are emitted by both HTTP/1.1 and HTTP/2 code paths unless noted otherwise. The prefix (default sozu) is omitted from metric names below.

Infrastructure

Metric	Type	Scope	Description
panic	counter	proxy	Worker thread panicked (logged before crash)
configuration.clusters	gauge	proxy	Number of configured clusters
configuration.backends	gauge	proxy	Number of configured backend servers
configuration.frontends	gauge	proxy	Number of configured frontends
client.connections	gauge	proxy	Active frontend connections
client.connections_max	gauge	proxy	Configured max_connections. Renamed from client.max_connections
client.connections_percent	gauge	proxy	Percentage of max_connections in use. Renamed from client.connections_percentage
connections.rejected_per_cluster_ip	counter	cluster, backend	HTTP/HTTPS request answered with 429 (or TCP session closed pre-backend) because the per-(cluster, source-IP) connection limit was reached. Labels carry cluster_id and backend_id (always empty for this counter — the rejection happens before backend selection).
slab.entries	gauge	proxy	Session slab allocator slots used
slab.capacity	gauge	proxy	Configured slab capacity (10 + slab_entries_per_connection * max_connections)
slab.usage_percent	gauge	proxy	slab.entries * 100 / slab.capacity. Pure slab-utilisation gauge
slab.accept_threshold_percent	gauge	proxy	slab.entries * 100 / (10 + 2 * max_connections). Charts proximity to the at_capacity() accept gate, which can flip true while slab.usage_percent still shows headroom (configured slab is larger when slab_entries_per_connection > 2)
process.uptime_seconds	gauge	proxy	Seconds since the worker started. Captured once in Server::new; never reset on hot upgrade (the new worker starts its own counter)
server.live	gauge	proxy	1 while the worker accepts traffic, 0 once a graceful shutdown is requested. Mirrors Envoy server.live semantics — L4 health checks (HAProxy / cloud LBs) can poll this gauge to drain a worker before the OS-level termination signal lands
buffer.in_use	gauge	proxy	Buffers currently checked out of the buffer pool. Renamed from buffer.number
buffer.capacity	gauge	proxy	Configured buffer pool capacity
buffer.usage_percent	gauge	proxy	buffer.in_use * 100 / buffer.capacity
zombies	counter	proxy	Zombie sessions detected and removed

Accept queue

Metric	Type	Scope	Description
accept_queue.connections	gauge	proxy	Sockets waiting in the accept queue
accept_queue.backpressure	gauge	proxy	1 when max connections reached, 0 when accepting again
accept_queue.wait_time	time	proxy	How long a socket waited in the accept queue (ms)
accept_queue.timeout	counter	proxy	Sockets that timed out in the accept queue and were closed
accept_queue.saturated_seconds	counter	proxy	Incremented at 1 Hz while SessionManager::can_accept is false. Distinguishes "queue spent N seconds at max" from "queue briefly hit max" — the binary accept_queue.backpressure gauge collapses that duration
listener.accepted.total	counter	proxy	Sockets accepted by the worker, all listeners combined
listener.accepted.tcp	counter	proxy	Sockets accepted on TCP listeners
listener.accepted.http	counter	proxy	Sockets accepted on HTTP listeners
listener.accepted.https	counter	proxy	Sockets accepted on HTTPS listeners
listener.connection_capped	counter	proxy	Sockets refused by create_sessions because SessionManager::check_limits returned false (max connections reached or slab at capacity)
client.connect.per_source.bucket_000 … bucket_255	counter	proxy	Per-accept counter bucketed by masked source subnet. Source IPs are masked to /24 (IPv4) or /48 (IPv6) and hashed (DefaultHasher) into 256 fixed buckets. The bucket noise is intentional: incr! requires &'static str keys, and a per-IP counter would be unbounded under SYN flood (OWASP A05, NIST SP 800-92). Operators wanting per-IP attribution should pair these counters with structured access logs or a downstream rate-limiter
sessions.evicted	counter	proxy	Sessions force-closed by evict_on_queue_full to make room for new accepts. Counts the number of evictions DECIDED in a cap event (one per call to evict_least_active_sessions); the helper may skip already-closed tokens, so a small drift versus the slab-removal count is possible. Only emitted when evict_on_queue_full = true

The accept-loop telemetry above does not label by listener address. incr! requires &'static str keys, and listener addresses can be added or removed at runtime via the control plane — labelling by address would either require runtime Box::leak of unbounded strings or a fixed bucket cap. We chose the per-protocol breakdown (3 keys + aggregate) instead. Operators needing per-listener-address attribution should run distinct workers per listener or correlate via access logs.

Cluster availability

Per-cluster signals that pair the existing EventKind::NoAvailableBackends with EventKind::ClusterRecovered. Latched on BackendList so the log + counter + event fire exactly once per Available ↔ AllDown transition, not once per request. Empty clusters (total == 0) emit the gauges but never log a transition — avoids spam during cluster bootstrap when backends are still being registered.

Metric	Type	Scope	Description
cluster.available_backends	gauge	cluster	Backends currently passing the availability predicate (status == Normal && health.is_healthy() && !retry_policy.is_down()). Updated on every routing decision and every health-check tick
cluster.total_backends	gauge	cluster	Backends configured for the cluster, regardless of state. Pairs with cluster.available_backends so dashboards can compute health ratios per cluster
cluster.no_available_backends	counter	cluster	Incremented exactly once per Available → AllDown transition. Pairs with the existing EventKind::NoAvailableBackends event and the error! log line cluster X: all N backends are down
cluster.available_recovered	counter	cluster	Incremented exactly once per AllDown → Available transition. Pairs with EventKind::ClusterRecovered (proto tag 29) and the info! log line cluster X: backends recovered (i/N available)
backend.available	gauge	backend	1 when the backend passes is_available() (health + retry policy + status), 0 after a transition to unavailable. Emitted at the up/down transition sites in health_check.rs, kawa_h1, mux, and tcp — not per-request, so the cardinality cost is bounded by transition frequency

The health_check.healthy_backends gauge is now labelled with cluster_id; prior emissions overwrote each other across clusters because the unlabelled key collapsed every cluster's value into a single bucket.

Works without an active health check

The tracker is driven by both the data-plane connect path and the active health-check tick. Clusters that have not configured a [clusters.<id>.health_check] block still see the full cluster.* and backend.available surface — transitions are detected through the per-backend retry policy as TCP connect attempts succeed or fail.

• Every TCP connect failure on the data path (lib/src/tcp.rs, lib/src/protocol/kawa_h1/mod.rs::fail_backend_connection, lib/src/protocol/mux/mod.rs) calls Backend::retry_policy.fail(), arming an exponential-backoff window. After max_tries consecutive failures (default 6) the policy reports is_down() == true. When every backend in the cluster reaches that state, the next routing call observes available == 0 && total > 0, flips the cell to AllDown, and emits the error! log line + cluster.no_available_backends counter + EventKind::NoAvailableBackends event.
• Every successful TCP connect calls Backend::retry_policy.succeed(), which clears is_down() immediately. The success arm of the routing call then re-evaluates and flips the cell back to Available, emitting the info! log line + cluster.available_recovered counter + EventKind::ClusterRecovered event.

Backends in the brief WAIT window (post-fail, pre-budget-exhaustion) are intentionally counted as available because the predicate is !retry_policy.is_down(), not Backend::can_open(). This avoids flapping AllDown ↔ Available when every backend happens to be in a 1-second backoff at the same instant. The WAIT state is transient by design; the cluster is still healthy from the operator's perspective, just briefly throttled. Only budget exhaustion (current_tries >= max_tries) drives the cell to AllDown.

Configuring an active health check on top of the data-plane signal adds a second observation cadence — useful when a cluster is idle (no requests, so no passive observations) but you still want timely up/down detection — but is not required for the per-cluster availability surface to function.

Event loop

Metric	Type	Scope	Description
epoll_time	time	proxy	Time spent in epoll_wait/kqueue (ms)
event_loop_time	time	proxy	Total event loop iteration time (ms)

Protocol state

These gauges track how many sessions are in each protocol phase. A session transitions through phases (e.g., Expect → TLS Handshake → HTTPS → WSS).

Metric	Type	Scope	Description
protocol.proxy.expect	gauge	proxy	Sessions expecting a PROXY protocol header
protocol.proxy.send	gauge	proxy	Sessions sending a PROXY protocol header to a backend
protocol.proxy.relay	gauge	proxy	Sessions relaying a PROXY protocol header
protocol.tls.handshake	gauge	proxy	Sessions in TLS handshake (HTTPS only)
protocol.http	gauge	proxy	Active HTTP sessions
protocol.https	gauge	proxy	Active HTTPS sessions
protocol.tcp	gauge	proxy	Active TCP proxy sessions
protocol.ws	gauge	proxy	Active WebSocket sessions (over HTTP)
protocol.wss	gauge	proxy	Active WebSocket sessions (over HTTPS)
websocket.active_requests	gauge	proxy	Active WebSocket requests (HTTP + HTTPS)

Request lifecycle

Metric	Type	Scope	Description
http.requests	counter	proxy, cluster, backend	Total HTTP requests received (incremented when headers are fully parsed)
http.active_requests	gauge	proxy	Currently in-flight requests
http.e2e.http11	counter	proxy	Completed HTTP/1.1 request/response cycles
http.e2e.h2	counter	proxy	Completed HTTP/2 request/response cycles
http.errors	counter	proxy, cluster, backend	General HTTP processing errors. Labels are filtered centrally per metrics.detail: process / frontend collapse to a proxy-wide counter, cluster (default) attributes per cluster, backend keeps the per-backend split
tcp.requests	counter	proxy	TCP proxy connection requests

Byte counters

Metric	Type	Scope	Description
bytes_in	counter	proxy, cluster, backend	Bytes received from frontend clients
bytes_out	counter	proxy, cluster, backend	Bytes sent to frontend clients
back_bytes_in	counter	proxy	Bytes received from backend servers
back_bytes_out	counter	proxy	Bytes sent to backend servers

Timing / latency

These are recorded as HDR histograms locally (queryable as percentiles: p50, p90, p99, p99.9, p99.99, p99.999, p100) and sent as |ms values over StatsD.

Metric	Type	Scope	Description
request_time	time	proxy, cluster	Total request time: first byte received to last byte sent (ms)
service_time	time	proxy, cluster	Internal processing time excluding backend I/O (ms)
backend_response_time	time	cluster, backend	Time from backend connection to last response byte (ms)
backend_connection_time	time	cluster, backend	TCP connection establishment time to backend (ms)
frontend_matching_time	time	cluster	Cluster/frontend route matching time (ms)
regex_matching_time	time	proxy	Regex evaluation time for path-based routing (ms)

Response status classes

Incremented per backend response (scope: cluster + backend for 1xx–5xx). Buckets are emitted for every response with a status; per-code counters below are emitted in addition to the bucket for the eighteen short-listed codes Sōzu either generates as a default answer or that operators routinely chart. Status codes outside that list contribute only to their bucket so the metric keyspace stays bounded.

Metric	Type	Scope	Description
http.status.1xx	counter	cluster, backend	1xx informational responses
http.status.2xx	counter	cluster, backend	2xx success responses
http.status.3xx	counter	cluster, backend	3xx redirection responses
http.status.4xx	counter	cluster, backend	4xx client error responses
http.status.5xx	counter	cluster, backend	5xx server error responses
http.status.other	counter	proxy	Non-standard status codes
http.status.none	counter	proxy	Responses without a status code
http.status.200 / 201 / 204	counter	cluster, backend	Common 2xx success codes (in addition to http.status.2xx)
http.status.301 / 302 / 304	counter	cluster, backend	Common 3xx redirect/cache codes (in addition to http.status.3xx)
http.status.400 / 401 / 403 / 404 / 408 / 413 / 429	counter	cluster, backend	Common 4xx client-error codes (in addition to http.status.4xx)
http.status.500 / 502 / 503 / 504 / 507	counter	cluster, backend	Common 5xx server-error codes (in addition to http.status.5xx)

Default answer / error responses

Incremented when Sōzu generates a default error response instead of proxying:

Metric	Type	Scope	Description
http.301.redirection	counter	proxy	301 Moved Permanently (HTTP→HTTPS redirect, RedirectPolicy::Permanent)
http.302.redirection	counter	proxy	302 Found (RedirectPolicy::Found — UA may rewrite POST → GET)
http.308.redirection	counter	proxy	308 Permanent Redirect (RedirectPolicy::PermanentRedirect — method MUST be preserved)
http.redirect_template.compile_error	counter	proxy	Operator-supplied per-frontend redirect_template failed to compile at request time; the listener default fired instead
http.400.errors	counter	proxy	400 Bad Request (cannot parse hostname)
http.401.errors	counter	proxy	401 Unauthorized
http.404.errors	counter	proxy	404 Not Found (no matching cluster)
http.408.errors	counter	proxy	408 Request Timeout
http.413.errors	counter	proxy	413 Payload Too Large
http.502.errors	counter	proxy	502 Bad Gateway
http.503.errors	counter	proxy	503 Service Unavailable (no backends or circuit breaker triggered)
http.504.errors	counter	proxy	504 Gateway Timeout
http.507.errors	counter	proxy	507 Insufficient Storage (buffer full)
http.other.errors	counter	proxy	Non-standard error response code (mux path only)

Parse errors

Metric	Type	Scope	Description
http.frontend_parse_errors	counter	proxy	Frontend request parsing failures (malformed HTTP/1.1 or HPACK decode errors in HTTP/2)
http.frontend.transfer_encoding_smuggling	counter	proxy	H1 request rejected (400) for ambiguous Transfer-Encoding framing: more than one non-elided Transfer-Encoding header, or one whose final coding is not chunked (e.g. chunked, gzip), or one present without kawa adopting chunked framing (RFC 9110 §7.6 / RFC 9112 §6.1; #726). Whitespace around a coding is not ambiguity — kawa >= 0.7.1 excludes it from the field value (RFC 9112 §5), so chunked\t frames as chunked and is forwarded canonically rather than counted here
http.backend_parse_errors	counter	proxy	Backend response parsing failures

Backend health

Metric	Type	Scope	Description
backend.connections	gauge	proxy	Active backend connections
connections_per_backend	gauge	cluster, backend	Per-backend connection count
backend.up	counter	proxy	Backend marked as healthy (after successful connection)
backend.down	counter	proxy	Backend marked as unhealthy (retry policy triggered)
backend.connections.error	counter	proxy	Backend connection failures
backend.connect.retries_exhausted	counter	cluster, backend	Per-session backend-connect retry budget (CONN_RETRIES = 3) was exhausted. Emitted once per event at the TCP, HTTP/1, and HTTP/2-mux gates. Alert on this counter's rate instead of grepping WARN / ERROR logs — the underlying log line is warn! since the condition is peer-driven backpressure, not a Sōzu invariant break.

Backend pool

H2 mux reuses backend connections via Router::backends: HashMap<Token, Connection> (lib/src/protocol/mux/router.rs). There is no separate pool abstraction: the map is the pool. Reuse picks an existing non-draining H2 multiplex slot (below SETTINGS_MAX_CONCURRENT_STREAMS) or an H1 keep-alive socket; misses dial a fresh backend socket.

Metric	Type	Scope	Description
backend.pool.hit	counter	proxy	Request attached to an existing backend connection (H2 multiplex slot or H1 keep-alive).
backend.pool.miss	counter	proxy	No reusable connection found; a fresh dial starts. Incremented before backend_from_request, so failed selections still count. Dial may still fail — in that case backend.pool.size is not bumped.
backend.pool.size	gauge	proxy	Live mux router entries. +1 at router.rs::connect new-dial commit; -1 at connection.rs::pre_close_client_bookkeeping and mod.rs::close_backend. Mirrors the backend.connections site set in mux exactly, so gauge symmetry follows from backend.connections correctness. Non-mux H1/TCP paths are NOT counted here.
backend.flow_control.paused	counter	proxy	Direction-scoped counterpart of h2.flow_control_stall: emitted only when the converter stalls while writing toward an upstream backend (Position::Client) because the backend's HTTP/2 receive window is empty.

Intentionally not emitted in this slice (no corresponding lifecycle exists):

• backend.pool.idle_closed — mux has no connection-level idle eviction. The H2 stream_idle_timeout cancels individual streams (slow-multiplex guard), not pool entries.
• backend.pool.overflow — Router::backends is unbounded. The only new-connection refusal is HTTP/2 buffer-pool exhaustion (MaxBuffers), unrelated to pool sizing.
• backend.flow_control.resumed — the converter has no "resumed" boundary; the next writable cycle just succeeds when the backend ACKs window updates. Plumbing an explicit marker through flush_stream_out was deferred.

Backend metrics (per cluster/backend)

These metrics are recorded with cluster_id and backend_id labels via the record_backend_metrics! macro at the end of each request:

Metric	Type	Scope	Description
requests	counter	cluster, backend	Requests handled by this backend
bytes_in	counter	cluster, backend	Bytes received from this backend
bytes_out	counter	cluster, backend	Bytes sent to this backend
backend_response_time	time	cluster, backend	Response time for this backend (ms)
backend_connection_time	time	cluster, backend	Connection setup time for this backend (ms)

ALPN negotiation

Metric	Type	Scope	Description
http.alpn.h2	counter	proxy	TLS connections where client negotiated HTTP/2 via ALPN
http.alpn.http11	counter	proxy	TLS connections where client negotiated HTTP/1.1 via ALPN (or no ALPN)
https.alpn.rejected.http11_disabled	counter	proxy	TLS connection refused on an H2-only listener because the peer offered http/1.1 or no ALPN value
https.alpn.rejected.unsupported	counter	proxy	TLS connection refused because the peer negotiated an ALPN protocol Sōzu does not recognise (anything other than h2, http/1.1, or absent ALPN). Add to the same SOC bucket as https.alpn.rejected.http11_disabled.

TLS version and cipher suite

Incremented once per TLS connection after the handshake completes.

Metric	Type	Scope	Description
tls.version.SSLv2	counter	proxy	Connections using SSLv2
tls.version.SSLv3	counter	proxy	Connections using SSLv3
tls.version.TLSv1_0	counter	proxy	Connections using TLS 1.0
tls.version.TLSv1_1	counter	proxy	Connections using TLS 1.1
tls.version.TLSv1_2	counter	proxy	Connections using TLS 1.2
tls.version.TLSv1_3	counter	proxy	Connections using TLS 1.3
tls.version.DTLSv1_0	counter	proxy	Connections using DTLS 1.0
tls.version.DTLSv1_2	counter	proxy	Connections using DTLS 1.2
tls.version.DTLSv1_3	counter	proxy	Connections using DTLS 1.3
tls.version.Unknown	counter	proxy	Unrecognized TLS version
tls.version.unimplemented	counter	proxy	TLS version not yet handled in code
tls.default_cert_used	counter	proxy	Fallback to default certificate (no SNI match)

Negotiated cipher suite (rustls):

Metric	Type	Scope
tls.cipher.TLS13_AES_128_GCM_SHA256	counter	proxy
tls.cipher.TLS13_AES_256_GCM_SHA384	counter	proxy
tls.cipher.TLS13_CHACHA20_POLY1305_SHA256	counter	proxy
tls.cipher.TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256	counter	proxy
tls.cipher.TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384	counter	proxy
tls.cipher.TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256	counter	proxy
tls.cipher.TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256	counter	proxy
tls.cipher.TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384	counter	proxy
tls.cipher.TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256	counter	proxy
tls.cipher.Unsupported	counter	proxy

TLS handshake telemetry

Emitted by the rustls handshake driver at lib/src/protocol/rustls.rs. Failure counters sit next to the tiered log emission in log_handshake_error so every warn!/error!/debug! about a broken handshake also bumps the matching counter. The histogram is recorded at the moment the handshake transitions out of is_handshaking() (both through the readable and writable exit paths).

Metric	Type	Scope	Description
tls.handshake.failed.alert_received	counter	proxy	Remote peer sent a fatal TLS alert (RustlsError::AlertReceived). Typical causes: cert-pinning client, stale CA bundle, scanner. Logged at debug!.
tls.handshake.failed.peer_incompatible	counter	proxy	Peer advertised a version/feature mix we cannot negotiate (PeerIncompatible). Logged at warn!.
tls.handshake.failed.peer_misbehaved	counter	proxy	Peer deviated from the TLS state machine (PeerMisbehaved). Logged at warn!.
tls.handshake.failed.invalid_message	counter	proxy	Wire-level record parse failure (InvalidMessage). Logged at warn!.
tls.handshake.failed.inappropriate_message	counter	proxy	Peer sent a record type that was valid on the wire but not allowed in the current phase (InappropriateMessage). Logged at warn!.
tls.handshake.failed.inappropriate_handshake_message	counter	proxy	Peer sent a handshake sub-type the state machine did not expect (InappropriateHandshakeMessage). Logged at warn!.
tls.handshake.failed.oversized_record	counter	proxy	Peer sent a record larger than the RFC 8446 §5.1 cap (PeerSentOversizedRecord). Logged at warn!.
tls.handshake.failed.no_alpn	counter	proxy	ALPN negotiation failed (NoApplicationProtocol) — e.g. peer offered only protocols the listener does not serve. Logged at warn!.
tls.handshake.failed.invalid_certificate	counter	proxy	Peer-supplied certificate failed verification (InvalidCertificate). Logged at warn!. Only relevant when mTLS is enabled.
tls.handshake.failed.decrypt_error	counter	proxy	Record failed to decrypt (DecryptError) — almost always an attack or a broken middlebox. Logged at warn!.
tls.handshake.failed.no_certificates_present	counter	proxy	mTLS: peer sent an empty Certificate message (NoCertificatesPresented). Logged at warn!.
tls.handshake.failed.other	counter	proxy	Catch-all for local/config/provider failures (General, Other, EncryptError, FailedToGetRandomBytes, CRL errors, future rustls variants). Logged at error! — these indicate a server-side problem, not a bad client.
tls.handshake_ms	time	proxy	Wall-clock duration in milliseconds from the first TLS byte observed on the socket to the handshake leaving is_handshaking(). Histogram — not a counter — so alert rules should use quantiles.

TLS certificate expiration

Metric	Type	Scope	Description
tls.cert.min_expires_at_seconds	gauge	proxy	Unix-seconds timestamp of the soonest-expiring certificate currently loaded in the CertificateResolver. Recomputed on every add/remove/replace at lib/src/tls.rs. Aggregate only — per-SNI granularity is intentionally omitted because statsd has no label support and the resolver can hold tens of thousands of names; operators query per-cert detail through the command API. Already-expired certificates clamp to 0, which dashboards should interpret as the "rotate now" alert condition.

HTTP/2 specific

Metric	Type	Scope	Description
h2.connection.active_streams	gauge	proxy	Aggregate count of open HTTP/2 streams across every live H2 connection on the worker. Emitted as a [gauge_add!] lifecycle delta from ConnectionH2::gauge_connection_state; the per-connection contribution is subtracted on connection drop, so the value sums correctly under multi-connection load.
h2.connection.window_bytes	gauge	proxy	Aggregate sum of available connection-level flow-control window bytes across every live H2 connection. Negative per-connection windows clamp to 0 — the aggregate measures available capacity, not deficit. Lifecycle-delta semantics as above.
h2.connection.pending_window_updates	gauge	proxy	Aggregate number of queued (un-flushed) per-stream WINDOW_UPDATE entries across every live H2 connection. Lifecycle-delta semantics as above.

Migration note (renamed metrics). The three keys above replace the earlier h2.connection_window, h2.active_streams, and h2.pending_window_updates gauges. The old keys had per-connection snapshot semantics implemented via absolute gauge!: under concurrent load every H2 connection clobbered the previous one's value, so the dashboard saw the last writer rather than the aggregate. The new keys emit lifecycle deltas via gauge_add!, so the value is the sum across all live H2 connections on the worker. Update dashboards and alerts accordingly — the new values typically rise into the hundreds or thousands under load instead of cycling 0…N.

| h2.flow_control_stall | counter | proxy | Converter stalled due to flow control | | h2.streams.reaped.idle_timeout | counter | proxy | Streams reaped by the per-stream bidirectional-idle guard (slow-multiplex Slowloris) | | h2.streams.reaped.window_stall | counter | proxy | Streams reaped by the outbound flow-control-stall guard (a buffered response or backend upload the peer will not drain — the HTTP/2 window-stall / WINDOW_UPDATE-drip vector) | | h2.streams.reaped.stall_budget | counter | proxy | Subset of window_stall: reaps where the stream dribbled progress below the 16 KiB cumulative-stall floor (the WINDOW_UPDATE-drip the budget closes) | | h2.close_with_active_streams | counter | proxy | H2 connections closed while streams were still active | | h2.window_update_dropped | counter | proxy | WINDOW_UPDATE frame dropped because the per-connection pending-update queue was already at capacity | | h2.headers_no_stream.error | counter | proxy | HEADERS frame received with no matching stream (protocol error) | | h2.frames.tx.headers | counter | proxy | HEADERS frames emitted by the H2 block converter (one per response, plus the first frame of any header block split into HEADERS+CONTINUATION when the encoded headers exceed the negotiated max_frame_size) | | h2.frames.tx.continuation | counter | proxy | CONTINUATION frames emitted by the converter when a single response's encoded headers cross max_frame_size (default 16 KB). Stays at zero for typical responses | | h2.frames.tx.data | counter | proxy | DATA frames emitted by the converter — both the normal data path and the empty END_STREAM-only marker emitted when the response has zero bytes after headers |

TLS / SNI binding

Metric	Type	Scope	Description
http.sni_authority_mismatch	counter	proxy	Request rejected because its :authority (HTTP/2) or Host header (HTTP/1.1) was not covered by any SAN dNSName of the certificate served on this TLS session (RFC 6125 §6.4.4, with RFC 6125 §6.4.3 wildcard handling). Default-cert handshakes fall back to legacy SNI exact-match. Defence against cross-tenant frontend confusion (CWE-346). Increment site: lib/src/protocol/mux/router.rs. The historical sni_* prefix is preserved so existing operator dashboards keep working.
h2.coalescing.accepted	counter	proxy	HTTP/2 stream accepted via RFC 7540 §9.1.1 / RFC 9113 §9.1.1 connection coalescing — :authority differed from the TLS SNI negotiated at handshake but matched a SAN dNSName of the served certificate (with RFC 6125 §6.4.3 wildcards). ALPN-gated to h2 so HTTP/1.1 cross-Host keep-alive does not double-count. Multi-tenant observability for Firefox / Chrome coalescing on shared wildcard certs. Increment site: lib/src/protocol/mux/router.rs.

HTTP/2 frame counters

Per-frame-type counters split by direction. Receive side counts every H2 frame the parser hands to handle_frame — single chokepoint, so adding a new H2 frame type fails the build inside the metric helper. Send side covers the control frames Sozu emits; HEADERS and DATA tx flow through the H2 block converter and are not yet broken out per type (tracked as follow-up — pair with back_bytes_out for the byte view today).

Metric	Type	Scope	Description
h2.frames.rx.data	counter	proxy	DATA frames received
h2.frames.rx.headers	counter	proxy	HEADERS frames received
h2.frames.rx.push_promise	counter	proxy	PUSH_PROMISE frames received (always rejected; see h2.goaway.sent.protocol_error)
h2.frames.rx.priority	counter	proxy	RFC 7540 PRIORITY frames received
h2.frames.rx.rst_stream	counter	proxy	RST_STREAM frames received (paired with h2.rst_stream.received.<code> for per-error breakdown)
h2.frames.rx.settings	counter	proxy	SETTINGS frames received (both peer-settings and ACKs)
h2.frames.rx.ping	counter	proxy	PING frames received (both probes and ACKs)
h2.frames.rx.goaway	counter	proxy	GOAWAY frames received (paired with h2.goaway.received.<code>)
h2.frames.rx.window_update	counter	proxy	WINDOW_UPDATE frames received
h2.frames.rx.continuation	counter	proxy	Reachable only via the defensive fallback path (RFC 9113 §6.10 standalone CONTINUATION) — the inline header parser absorbs CONTINUATION during HEADERS decoding, so under normal conditions this stays at zero
h2.frames.rx.unknown	counter	proxy	Unknown frame type ignored per RFC 9113 §5.5. A non-zero rate is the early-warning signal for a peer trying H2 extensions
h2.frames.tx.settings	counter	proxy	SETTINGS frames emitted (initial + later updates)
h2.frames.tx.window_update	counter	proxy	WINDOW_UPDATE frames emitted (per-frame in the queued-update flush loop)
h2.frames.tx.rst_stream	counter	proxy	RST_STREAM frames emitted (across both the queued flush loop and the end-stream cancel path)
h2.frames.tx.goaway	counter	proxy	GOAWAY frames emitted (one per phase of graceful_goaway, plus error-path goaway)
h2.frames.tx.ping_ack	counter	proxy	PING ACK frames emitted in response to peer probes

HTTP/2 edge-triggered wake-up instrumentation

LIFECYCLE §9 invariant 15 compliance counters. Each firing records a site where sozu paired Ready::WRITABLE with signal_pending_write so the edge-triggered epoll scheduler re-runs writable() on the next tick — required whenever bytes land in sozu-owned buffers (the kernel never signals WRITABLE for buffers it does not own). Useful as a regression baseline: the first two correlate with 504/500/400 rate and RFC 9218 PRIORITY_UPDATE rate respectively, the last two correlate with backend-H2 traffic volume. A 10× spike on any counter without a matching spike on the upstream driver is the early warning for a hot loop regression.

Metric	Type	Scope	Description
h2.signal.writable.rearmed.default_answer	counter	proxy	Fired in mux/answers.rs::set_default_answer (504 backend timeout, 500/400 parse errors). Rate should track the default-answer render rate closely.
h2.signal.writable.rearmed.forcefully_terminate_answer	counter	proxy	Fired in mux/answers.rs::forcefully_terminate_answer when the proxy injects a default answer mid-response (e.g. backend disconnects after partial body). Companion to default_answer; the forcefully-terminate path is exercised on backend hard-failure and proxy-initiated stream resets.
h2.signal.writable.rearmed.priority_update	counter	proxy	Fired in mux/h2.rs::handle_priority_update_frame whenever a PRIORITY_UPDATE mutates Prioriser state. Pairs with h2.frames.rx.priority_update. Low-to-zero on Firefox-only fleets (Firefox does not emit 0x10).
h2.signal.writable.rearmed.peer_data	counter	proxy	Fired in mux/h2.rs::handle_data_frame when an H2 DATA frame wakes the linked peer. Non-zero only on clusters that use H2 to the origin.
h2.signal.writable.rearmed.peer_headers	counter	proxy	Fired in mux/h2.rs::handle_headers_frame when an H2 HEADERS frame wakes the linked peer. Non-zero only on clusters that use H2 to the origin.
h2.signal.writable.rearmed.control_queue	counter	proxy	Fired in mux/h2.rs::flush_pending_control_frames when a queued WINDOW_UPDATE or RST_STREAM forces an extra writable pass. Pairs with h2.frames.tx.window_update / h2.frames.tx.rst_stream. Persistent non-zero rate without matching tx growth points to a control-frame queue that fills faster than it drains.
h2.streams.ready_incremental.by_urgency	gauge	proxy	Post-scheduling-pass snapshot of the sum of ready incremental streams across all urgency buckets (RFC 9218 §4). Debug hint for scheduler fairness — a consistently non-zero value under load means the round-robin is active.
h2.trailers_dropped_content_length	counter	proxy	Fired in mux/pkawa.rs::handle_trailer when an H2 trailer block is rejected for carrying a Content-Length header (RFC 9113 §8.1 disallows pseudo-headers and Content-Length in trailers). Spikes correlate with malformed gRPC clients or smuggling attempts; aggregate cardinality is bounded.
h1.backend_eof_before_message_complete	counter	proxy	Fired in mux/h1.rs::back_readable when the H1 backend closes the socket before the response body is fully delivered (chunked-EOF or Content-Length-EOF mid-stream). The H2 converter surfaces this as RST_STREAM(InternalError) to the H2 client; non-zero rate maps to backend application crashes or proxy-side reads.

HTTP/2 GOAWAY and RST_STREAM by error code

Counters split by direction (sent vs received) and RFC 9113 §7 error code. Every variant in the H2Error enum gets its own counter so SOC dashboards can distinguish a sustained protocol_error (parser issue / fuzzer) from enhance_your_calm (flood-detector trip) from no_error (graceful drain). Codes the wire delivers but RFC 9113 does not define are bucketed under unknown_error to keep cardinality bounded.

Metric pattern	Type	Scope	Description
h2.goaway.sent.<code>	counter	proxy	GOAWAY emitted by Sozu. <code> ∈ no_error, protocol_error, internal_error, flow_control_error, settings_timeout, stream_closed, frame_size_error, refused_stream, cancel, compression_error, connect_error, enhance_your_calm, inadequate_security, http_1_1_required. The graceful drain (graceful_goaway) emits two no_error increments per connection — one per phase.
h2.goaway.received.<code>	counter	proxy	GOAWAY received from peer. Same code suffixes as sent, plus unknown_error for codes outside RFC 9113 §7. Useful on backend H2 connections to detect upstreams under pressure (enhance_your_calm) or with bugs (internal_error).
h2.rst_stream.sent.<code>	counter	proxy	RST_STREAM emitted by Sozu. Same code suffixes. The no_error and cancel increments are graceful (stream recycle, propagated client cancel); the rest are server-side defences against attacker-crafted frames.
h2.rst_stream.received.<code>	counter	proxy	RST_STREAM received from peer. Same code suffixes plus unknown_error.
h2.rst_stream.received.pre_response_start	counter	proxy	Subset of h2.rst_stream.received.* where the RST arrived before the backend started answering. The canonical Rapid Reset signature (CVE-2023-44487). Emitted alongside the per-code counter, not instead of, so a Rapid Reset attack surfaces both as a cancel rate spike and as the pre-response signal.

HTTP/2 header validation (HPACK rejections)

Counters emitted whenever the HPACK decoder in the H2→H1 converter rejects a header. h2.headers.rejected.total is bumped on every reject; a per-reason counter is bumped alongside so total == sum(per_reason). Rejection is silent on the wire (the mux either RSTs the stream or treats the request as malformed) — this counter family is the only externally visible signal for request-smuggling probes, HPACK fuzzing, and H2-specific protocol abuse.

Metric	Type	Scope	Description
h2.headers.rejected.total	counter	proxy	Aggregate count of all HPACK rejections
h2.headers.rejected.invalid_name_byte	counter	proxy	Header name with uppercase / CTL / separator / space byte (CWE-93)
h2.headers.rejected.connection_specific_header	counter	proxy	connection, proxy-connection, transfer-encoding, upgrade, keep-alive (RFC 9113 §8.2.2)
h2.headers.rejected.te_not_trailers	counter	proxy	te with value other than trailers (RFC 9113 §8.2.2)
h2.headers.rejected.crlf_in_value	counter	proxy	CR or LF in a header value — request-smuggling vector (CWE-444)
h2.headers.rejected.nul_in_value	counter	proxy	NUL byte in a header value
h2.headers.rejected.oversized_pseudo_value	counter	proxy	Pseudo-header value exceeded storage cap
h2.headers.rejected.cl_te_conflict	counter	proxy	Content-Length declared alongside a Transfer-Encoding — classic smuggling vector
h2.headers.rejected.duplicate_cl	counter	proxy	Multiple disagreeing Content-Length values (RFC 9110 §8.6)
h2.headers.rejected.duplicate_pseudo	counter	proxy	Same pseudo-header appeared twice
h2.headers.rejected.pseudo_after_regular	counter	proxy	Pseudo-header appeared after a regular header (RFC 9113 §8.3)
h2.headers.rejected.unknown_pseudo	counter	proxy	Unknown : -prefixed pseudo-header
h2.headers.rejected.empty_pseudo	counter	proxy	Pseudo-header value was empty (RFC 9113 §8.3.1)
h2.headers.rejected.invalid_method	counter	proxy	:method value was not a valid RFC 9110 §9 token
h2.headers.rejected.invalid_scheme	counter	proxy	:scheme was not http or https
h2.headers.rejected.invalid_path	counter	proxy	:path contained a # fragment (not allowed on the wire)
h2.headers.rejected.invalid_status	counter	proxy	Response :status was not three ASCII digits
h2.headers.rejected.header_list_size	counter	proxy	HEADERS/trailer block exceeded SETTINGS_MAX_HEADER_LIST_SIZE (name + value + 32 octets/field, RFC 9113 §6.5.2) — the indexed-reference "header bomb" byte budget. Block-level reject (ENHANCE_YOUR_CALM)
h2.headers.rejected.header_fields	counter	proxy	HEADERS/trailer block exceeded h2_max_header_fields materialized fields (cookie crumbs counted individually, RFC 9113 §8.2.3). Block-level reject (ENHANCE_YOUR_CALM)

Request-ID propagation

Sōzu preserves or generates an x-request-id header on every H1 request and every H2 stream (both paths share the same H1 editor callback via pkawa.rs). The value also lands on the access log's x_request_id field — same value Sōzu forwarded to the backend, end to end.

Metric	Type	Scope	Description
http.x_request_id.propagated	counter	proxy	Request already carried an x-request-id header; Sōzu preserved it verbatim
http.x_request_id.generated	counter	proxy	Request had no x-request-id; Sōzu generated one from the request ULID and injected it before forwarding

The x_request_id access-log field (wire tag ProtobufAccessLog.x_request_id #24) carries whichever value was sent to the backend.

TLS handshake metadata on access logs

Five additional access-log fields surface the negotiated TLS metadata and the upstream-attested forwarded chain. They are wire-compatible appends to ProtobufAccessLog (tags 25–29) and are populated end-to-end on H1 and H2 mux paths, plus the WSS post-upgrade pipe. Pure plaintext paths (HTTP, WS, TCP) emit None for all five.

Field	Wire tag	Source	Notes
tls_version	ProtobufAccessLog.tls_version #25	rustls_version_label(handshake.session.protocol_version())	Short form (e.g. TLSv1.3). Captured once at handshake completion in lib/src/https.rs::upgrade_handshake. None when rustls reports an unknown variant.
tls_cipher	ProtobufAccessLog.tls_cipher #26	rustls_ciphersuite_label(handshake.session.negotiated_cipher_suite())	Short form (e.g. TLS_AES_128_GCM_SHA256). None when rustls reports an unsupported cipher.
tls_sni	ProtobufAccessLog.tls_sni #27	handshake.session.server_name()	Pre-lowercased, no port. Same value the routing layer uses to enforce the SNI ↔ :authority binding. None when the client omitted SNI.
tls_alpn	ProtobufAccessLog.tls_alpn #28	ALPN negotiation in upgrade_handshake	h2, http/1.1, or None when no ALPN was negotiated.
xff_chain	ProtobufAccessLog.xff_chain #29	Verbatim X-Forwarded-For header value	Snapshotted in editor.rs::on_request_headers before Sōzu appends its own peer hop, so the log records the upstream-attested chain (e.g. 203.0.113.5, 198.51.100.10). None when the request has no X-Forwarded-For header.

TLS fields are connection-scoped: they are stamped once on the mux Context at handshake time and propagated to every per-stream HttpContext via Context::create_stream, so an H2 connection multiplexing N streams pays the cost once. The labels are &'static str borrows into rustls's static label tables — no per-request allocation.

Round-trip-time fields on access logs

client_rtt and server_rtt carry the kernel-measured TCP round-trip time on each side of the proxy at the moment the access log is emitted. Source: getsockopt(TCP_INFO) (Linux SOL_TCP, BSD IPPROTO_TCP, Darwin IPPROTO_TCP opt 0x106) wrapped in lib/src/socket.rs::stats::socket_rtt. Unit on the wire: microseconds (uint64); on Darwin the kernel reports tcpi_srtt in milliseconds and the helper multiplies by 1000 before exposing the same Duration.

Access-log field	Wire tag	Populated on
client_rtt	ProtobufAccessLog.client_rtt #9 (optional uint64)	every protocol path: H1 (kawa_h1), H2 (mux), Pipe (TCP/WS), TCP frontend
server_rtt	ProtobufAccessLog.server_rtt #10 (optional uint64)	every protocol path that has a backend socket; None for the TCP frontend (no upstream)

Capture is at access-log emission time and is cheap (one getsockopt(TCP_INFO) syscall per side), so the cell reflects the most recent kernel SRTT estimate rather than a session-wide average. None on AF_UNIX or any FSM state where TCP_INFO is not usable (pre-handshake, dead socket). Real implementations exist for Linux, FreeBSD, NetBSD, OpenBSD, DragonFly, macOS and iOS; non-Unix stub builds short-circuit to None.

HTTP/2 flood mitigations

Incremented once per connection at the moment the H2 flood detector trips its threshold and the proxy escalates to GOAWAY(ENHANCE_YOUR_CALM). Every CVE mitigation in the H2 family (Rapid Reset, MadeYouReset, the CONTINUATION / PING / SETTINGS / empty-DATA flood family, oversized header lists, and the generic glitch budget) routes through ConnectionH2::handle_flood_violation, which emits both the contextual log line and the per-kind counter below.

Metric	Type	Scope	Description
h2.flood.violation.rst_stream_window	counter	proxy	Per-window RST_STREAM rate ceiling exceeded. Generic stream-cancel storm signal — usually a misbehaving client, sometimes Rapid Reset.
h2.flood.violation.rst_stream_lifetime	counter	proxy	Lifetime received-RST ceiling exceeded. Catches a patient Rapid Reset attacker that stays under the windowed cap (CVE-2023-44487).
h2.flood.violation.rst_stream_pre_response_lifetime	counter	proxy	Lifetime received-RST ceiling exceeded for streams that the backend had not yet started answering. The canonical Rapid Reset signature (CVE-2023-44487).
h2.flood.violation.rst_stream_emitted_lifetime	counter	proxy	Lifetime server-emitted RST ceiling exceeded. MadeYouReset mitigation (CVE-2025-8671) — peer kept feeding the server crafted frames that forced it to reset streams.
h2.flood.violation.ping_window	counter	proxy	Per-window PING flood (CVE-2019-9512).
h2.flood.violation.ping_lifetime	counter	proxy	Lifetime PING ceiling exceeded — catches sustained low-rate PING abuse that stays under the windowed cap.
h2.flood.violation.settings_window	counter	proxy	Per-window SETTINGS flood (CVE-2019-9515).
h2.flood.violation.settings_lifetime	counter	proxy	Lifetime SETTINGS ceiling exceeded.
h2.flood.violation.empty_data_window	counter	proxy	Per-window flood of empty DATA frames (CVE-2019-9518).
h2.flood.violation.continuation_per_block	counter	proxy	Single header block split across more CONTINUATION frames than the configured cap (CVE-2024-27316).
h2.flood.violation.header_size_per_block	counter	proxy	Single header block accumulated more bytes than the configured cap (CVE-2024-27316 sibling — header overflow).
h2.flood.violation.glitch_window	counter	proxy	Generic anomaly budget exceeded (unknown SETTINGS, WINDOW_UPDATE on closed stream, other low-severity protocol drift).

Protocol upgrade failures

Incremented when a session fails to transition between protocol phases:

Metric	Type	Scope	Description
http.upgrade.expect.failed	counter	proxy	HTTP: PROXY protocol expect → mux transition failed
http.upgrade.mux.failed	counter	proxy	HTTP: mux protocol upgrade failed
http.upgrade.ws.failed	counter	proxy	HTTP: WebSocket upgrade failed
https.upgrade.expect.failed	counter	proxy	HTTPS: PROXY protocol expect → handshake transition failed
https.upgrade.handshake.failed	counter	proxy	HTTPS: TLS handshake → mux transition failed
https.upgrade.mux.failed	counter	proxy	HTTPS: mux protocol upgrade failed
https.upgrade.wss.failed	counter	proxy	HTTPS: WebSocket over TLS upgrade failed
tcp.upgrade.pipe.failed	counter	proxy	TCP: pipe protocol upgrade failed
tcp.upgrade.send.failed	counter	proxy	TCP: PROXY protocol send transition failed
tcp.upgrade.relay.failed	counter	proxy	TCP: PROXY protocol relay transition failed
tcp.upgrade.expect.failed	counter	proxy	TCP: PROXY protocol expect transition failed
tcp.upgrade.sni_preread.failed	counter	proxy	TCP: SNI-preread → pipe / PROXY-protocol-send transition failed (see below)

TCP SNI preread (passthrough routing)

Emitted by the SNI+ALPN preread routing feature (#1279) on a TCP listener with at least one hostname-scoped frontend. tcp.sni_preread.active is a gauge and is guaranteed not to underflow: it is incremented exactly once per session entering the preread state and decremented exactly once on whichever of its three mutually-exclusive exits fires (reject / upgrade-to-backend / teardown). A bare TCP health check (connect then close, zero bytes sent) is not counted in any rejected.* reason — see the reject-policy note in Options specific to TCP listeners.

Metric	Type	Scope	Description
tcp.sni_preread.routed	counter	proxy	A cluster was chosen from the ClientHello's SNI (+ optional ALPN)
tcp.sni_preread.active	gauge	proxy	Sessions currently prereading. Never underflows
tcp.sni_preread.duration	time	proxy	Wall-clock time spent prereading, recorded on every exit (including the defensive upgrade-time early-return paths, which record it immediately rather than waiting on the active decrement, since the latter may only land later, in close())
tcp.sni_preread.rejected.not_tls	counter	proxy	First TLS record's ContentType wasn't handshake
tcp.sni_preread.rejected.malformed_record	counter	proxy	Bad TLS record framing (declared length lies, or a non-handshake record interrupts an in-progress hello)
tcp.sni_preread.rejected.malformed_handshake	counter	proxy	Not a ClientHello, or a length-prefixed field inside it lies about the bytes available
tcp.sni_preread.rejected.fragmented	counter	proxy	The preread deadline (sni_preread_timeout) fired before a terminal verdict
tcp.sni_preread.rejected.too_large	counter	proxy	The ClientHello failed to complete within sni_preread_max_bytes (the preread never reads past the cap; a hello that completes within it routes even when coalesced bytes follow)
tcp.sni_preread.rejected.no_sni	counter	proxy	The server_name extension was absent, empty, or its host_name entry missing
tcp.sni_preread.rejected.ech_outer_absent	counter	proxy	Encrypted Client Hello (0xfe0d) was present with no usable outer SNI — distinct from no_sni
tcp.sni_preread.rejected.sni_unmatched	counter	proxy	The normalized SNI matched no configured hostname on this listener
tcp.sni_preread.rejected.alpn_unmatched	counter	proxy	The SNI matched a route but no entry's alpn accepted the client's offer (and no catch-all was present)
tcp.sni_preread.rejected.proxy_header_invalid	counter	proxy	The inbound PROXY-v2 header (on an expect_proxy listener) failed to parse
tcp.sni_preread.rejected.front_closed	counter	proxy	The frontend closed before a decision was reached (bytes were seen; a bare zero-byte health check is silent, not counted here)

See lib/src/protocol/tcp_preread/LIFECYCLE.md for the full state lifecycle, the four proxy_protocol handoff paths out of preread, and the reasoning behind each reject reason.

Socket and I/O errors

Metric	Type	Scope	Description
socket.read.infinite_loop.error	counter	proxy	TCP socket read loop safety breaker triggered
socket.write.infinite_loop.error	counter	proxy	TCP socket write loop safety breaker triggered
tcp.read.error	counter	proxy	TCP socket read error
tcp.write.error	counter	proxy	TCP socket write error
tcp.infinite_loop.error	counter	proxy	TCP session event loop safety breaker triggered
rustls.read.error	counter	proxy	TLS read error
rustls.write.error	counter	proxy	TLS write error
rustls.read.infinite_loop.error	counter	proxy	TLS read loop safety breaker triggered
rustls.write.infinite_loop.error	counter	proxy	TLS write loop safety breaker triggered

UDP

Emitted by protocol = "udp" listeners. udp.active_flows is a gauge and is guaranteed not to underflow on close / timeout / shed / error paths.

Metric	Type	Scope	Description
udp.datagrams.in	counter	cluster, backend	Client datagrams received
udp.datagrams.out	counter	cluster, backend	Datagrams forwarded (to backend) and returned (to client)
udp.bytes.in	counter	cluster, backend	Bytes received from clients
udp.bytes.out	counter	cluster, backend	Bytes sent (to backend and back to client)
udp.active_flows	gauge	proxy	Currently tracked flows (one connected upstream socket each). Never underflows
udp.flows.created	counter	cluster, backend	Flows admitted and tracked
udp.flows.evicted	counter	cluster, backend	Flows torn down (idle timeout / responses reached / requests reached / drain)
udp.flows.shed	counter	proxy	New flows dropped at the max_flows cap or on EMFILE/ENFILE (existing flows protected)
udp.datagrams.dropped	counter	proxy	Datagrams dropped, by reason: invalid / truncated / no-backend / shed / wq-full (write-queue full) / unknown-flow / send-error
udp.backend.health	gauge	cluster, backend	1 when the backend passes its UDP health check, 0 when down
udp.flow.duration	time	cluster, backend	Flow lifetime, from admission to teardown

Other

Metric	Type	Scope	Description
http.infinite_loop.error	counter	proxy	HTTP event loop safety breaker triggered
http.failed_backend_matching	counter	proxy	Frontend matched but no backend could be selected
http.early_response_close	counter	proxy	Client closed before response was fully sent
http.trusting.x_proto	counter	proxy	Request had an existing X-Forwarded-Proto header (trusted)
http.trusting.x_proto.diff	counter	proxy	Trusted X-Forwarded-Proto differed from actual protocol
http.trusting.x_port	counter	proxy	Request had an existing X-Forwarded-Port header (trusted)
http.trusting.x_port.diff	counter	proxy	Trusted X-Forwarded-Port differed from actual port
pipe.errors	counter	proxy	Pipe/WebSocket protocol errors
proxy_protocol.errors	counter	proxy	PROXY protocol v1/v2 parsing errors
unsent-access-logs	counter	proxy	Access log entries that could not be sent
access_logs.count	counter	cluster, backend	Access log entries emitted per cluster/backend

Example of external services

• statsd
• grad

OpenTelemetry (traceparent passthrough)

The opentelemetry compile-time feature flag enables W3C Trace Context passthrough: Sōzu parses, generates, and forwards traceparent headers across the proxy hop, and records the trace identifiers in its access logs. The feature name is historic — the implementation is intentionally a propagator only. There is no OpenTelemetry SDK dependency, no span lifecycle, and no OTLP exporter. See "Out of scope" below.

Enabling

Build Sōzu with the opentelemetry feature:

cargo build --release --features opentelemetry

Or in Cargo.toml:

[dependencies]
sozu-lib = { path = "lib", features = ["opentelemetry"] }

How it works

When the opentelemetry feature is enabled, Sōzu acts as a trace context propagator for both HTTP/1.1 and HTTP/2 frontends. The kawa H1 editor (lib/src/protocol/kawa_h1/editor.rs) runs on every request, including those decoded from HPACK on the H2 mux path (lib/src/protocol/mux/pkawa.rs calls back into the same editor). On the wire the H2 backend leg re-encodes the rewritten traceparent via H2BlockConverter.

1. Incoming request with traceparent header: Sōzu parses the W3C traceparent (format: 00-<trace_id>-<parent_id>-<flags>), preserves the trace ID, generates a new span ID for the Sōzu hop, and rewrites the header before forwarding to the backend.

2. Incoming request without traceparent header: Sōzu generates a new random trace ID and span ID, and injects a traceparent header into the request before forwarding.

3. tracestate header: Preserved if a valid traceparent is present. Elided if no traceparent accompanies it (per W3C spec).

4. Access logs: The trace context (trace ID, span ID, parent span ID) is included in access log entries and in the protobuf AccessLog message, enabling correlation between Sōzu access logs and distributed traces in your observability platform.

Access log fields

When OpenTelemetry is enabled, access logs include:

Field	Format	Description
trace_id	32 hex characters	W3C trace ID (propagated or generated)
span_id	16 hex characters	Sōzu-generated span ID for this hop
parent_span_id	16 hex characters or -	Parent span ID from incoming traceparent, if present

Out of scope

The opentelemetry feature is intentionally narrow. It does NOT:

• Pull opentelemetry, opentelemetry-sdk, opentelemetry-otlp, or tracing-opentelemetry as dependencies. The feature gate (lib/Cargo.toml) wires no extra crates.
• Emit spans. There is no Span::start/Span::end, no in-process exporter, no OTLP/gRPC client. Use your access log pipeline to feed Jaeger, Tempo, Honeycomb, or Datadog with the per-request trace identifiers.
• Honour the sampled-flag bit of incoming traceparent. Today the rewritten header is always emitted with -01 (sampled). Downstream backends that respect the sampled flag will see every request as sampled.
• Provide runtime configuration. The feature is selected at compile time only.

A real span model and an OTLP exporter are tracked separately and would land as their own feature flag rather than expanding the scope of opentelemetry.

PROXY Protocol

When a network stream goes through a proxy, the backend server will only see the IP address and port used by the proxy as client address. The real source IP address and port will only be seen by the proxy. Since this information is useful for logging, security, etc, the PROXY protocol was developed to transmit it to backend servers. With this protocol, after connecting to the backend server, the proxy will first send a small header indicating the client IP address and port, and the proxy's receiving IP address and port, and will then send the stream from the client.

Sōzu support the version 2 of the PROXY protocol in three configurations:

• "send" protocol: Sōzu, in TCP proxy mode, will send the header to the backend server
• "expect" protocol: Sōzu receives the header from a proxy, interprets it for its own logging and metrics, and uses it in HTTP forwarding headers
• "relay" protocol: Sōzu, in TCP proxy mode, can receive the header, and transmit it to a backend server

More information here: proxy-protocol spec

Configuring Sōzu to expect a PROXY Protocol header

Configures the client-facing connection to receive a PROXY protocol header before any byte sent by the client is read from the socket.

                           send PROXY                    expect PROXY
                           protocol header               protocol header
    +--------+
    | client |             +---------+                   +------------+      +-----------+
    |        |             | proxy   |                   | Sozu       |      | upstream  |
    +--------+  ---------> | server  |  ---------------> |            |------| server    |
   /        /              |         |                   |            |      |           |
  /________/               +---------+                   +------------+      +-----------+

It is supported by HTTP, HTTPS and TCP proxies.

Configuration:

[[listeners]]
address = "0.0.0.0:80"
expect_proxy = true

Configuring Sōzu to send a PROXY Protocol header to an upstream backend

Send a PROXY protocol header over any connection established to the backends declared in the cluster.

                           send PROXY
    +--------+             protocol header
    | client |             +---------+                +-----------------+
    |        |             | Sozu    |                | proxy/upstream  |
    +--------+  ---------> |         |  ------------> | server          |
   /        /              |         |                |                 |
  /________/               +---------+                +-----------------+

Configuration:

[[listeners]]
address = "0.0.0.0:81"

[clusters]
[clusters.NameOfYourTcpCluster]
send_proxy = true
frontends = [
  { address = "0.0.0.0:81" }
]

NOTE: Only for TCP clusters (HTTP and HTTPS proxies will use the forwarding headers).

Configuring Sōzu to relay a PROXY Protocol header to an upstream

Sōzu will receive a PROXY protocol header from the client connection, check its validity and then forward it to an upstream backend. This allows for chains of reverse-proxies without losing the client connection information.

                           send PROXY                       expect PROXY               send PROXY
                           protocol header                  protocol header            protocol header
    +--------+
    | client |             +---------+                      +------------+             +-------------------+
    |        |             | proxy   |                      | Sozu       |             | proxy/upstream    |
    +--------+  +--------> | server  |  +-----------------> |            | +---------> | server            |
   /        /              |         |                      |            |             |                   |
  /________/               +---------+                      +------------+             +-------------------+

Configuration:

This only concerns TCP clusters (HTTP and HTTPS proxies can work directly in expect mode, and will use the forwarding headers).

[[listeners]]
address = "0.0.0.0:80"
expect_proxy = true

[clusters]

[clusters.NameOfYourCluster]
send_proxy = true
frontends = [
  { address = "0.0.0.0:80" }
]