feat: add RouteUpstreamLimitsFilter for per-service upstream limits and passive health checking by bharath-k1999 · Pull Request #5596 · hashicorp/consul-k8s

bharath-k1999 · 2026-08-14T08:22:29Z

consul-k8s PR #5596 — API Gateway: Passive Health Checks & Upstream Limits

Summary

This PR adds Kubernetes-native support for configuring upstream circuit-breaker limits and passive health checks (Envoy outlier detection) on API Gateway backends.

Two configuration surfaces are exposed:

Surface	Scope	Mechanism
`RouteUpstreamLimitsFilter` CRD	Per-service (per backendRef)	`HTTPRoute` `extensionRef` filter
Gateway annotations	Gateway-wide defaults (all upstreams)	`api-gateway.consul.hashicorp.com/default-*` annotations

Service-level RouteUpstreamLimitsFilter values always override gateway-wide defaults for the matched backend.

What is implemented

New CRD: `RouteUpstreamLimitsFilter`

apiVersion: consul.hashicorp.com/v1alpha1
kind: RouteUpstreamLimitsFilter
metadata:
  name: static-server-limits
  namespace: default
spec:
  # Envoy circuit-breaker thresholds
  maxConnections: 25
  maxPendingRequests: 50
  maxConcurrentRequests: 100
  # Passive health check (Envoy outlier detection)
  passiveHealthCheck:
    interval: 10s
    maxFailures: 3
    enforcingConsecutive5xx: 100
    maxEjectionPercent: 50
    baseEjectionTime: 30s

Attach it to a specific backend via HTTPRoute extensionRef:

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: limits-route
spec:
  parentRefs:
    - name: my-gateway
  rules:
    - backendRefs:
        - name: static-server
          port: 80
          filters:
            - type: ExtensionRef
              extensionRef:
                group: consul.hashicorp.com
                kind: RouteUpstreamLimitsFilter
                name: static-server-limits

Gateway-wide defaults via annotations

Set default upstream limits on every backend behind a Gateway:

apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: my-gateway
  annotations:
    api-gateway.consul.hashicorp.com/default-max-connections: "40"
    api-gateway.consul.hashicorp.com/default-max-pending-requests: "80"
    api-gateway.consul.hashicorp.com/default-max-concurrent-requests: "160"
    api-gateway.consul.hashicorp.com/default-passive-health-check-interval: "15s"
    api-gateway.consul.hashicorp.com/default-passive-health-check-max-failures: "5"
    api-gateway.consul.hashicorp.com/default-passive-health-check-max-ejection-percent: "20"

These land in APIGatewayConfigEntry.Defaults in Consul and propagate to every upstream cluster via xDS.

Files changed

File	Change
`control-plane/api/v1alpha1/routeupstreamlimitsfilter_types.go`	CRD type (`RouteUpstreamLimitsFilter`, `RouteUpstreamLimitsFilterSpec`, `PassiveHealthCheck`)
`control-plane/api/v1alpha1/zz_generated.deepcopy.go`	Generated DeepCopy methods
`control-plane/config/crd/bases/consul.hashicorp.com_routeupstreamlimitsfilters.yaml`	CRD manifest
`charts/consul/templates/crd-routeupstreamlimitsfilters.yaml`	Helm chart CRD
`charts/consul/templates/connect-inject-clusterrole.yaml`	RBAC: `get/list/watch` on `routeupstreamlimitsfilters`
`control-plane/api-gateway/common/translation.go`	`translateGatewayDefaults`, `toConsulUpstreamLimits`, `translateBackendRefLimits`, `toConsulPassiveHealthCheck`
`control-plane/api-gateway/common/diff.go`	Bug fix: `Defaults` in `apiGatewaysEqual`; bug fix: `Limits` in `httpServicesEqual`; new `upstreamLimitsEqual` + `passiveHealthCheckEqual` helpers
`control-plane/api-gateway/controllers/gateway_controller.go`	Bug fix: split malformed `.Watches()` call; added `transformRouteUpstreamLimitsFilter`
`control-plane/api-gateway/controllers/index.go`	`HTTPRoute_RouteUpstreamLimitsFilterIndex` index for watch resolution
`control-plane/api-gateway/common/translation_upstreamlimits_test.go`	Unit tests for translation functions
`control-plane/api-gateway/common/diff_test.go`	New: `TestEntriesEqual_APIGateway_Defaults` (7 cases), `TestEntriesEqual_HTTPRoute_Limits` (7 cases)
`acceptance/tests/api-gateway/api_gateway_upstream_limits_test.go`	Acceptance test (6 sub-tests, see below)
`acceptance/tests/fixtures/cases/api-gateways/upstream-limits/`	Kustomize fixtures (kustomization, GatewayClassConfig, GatewayClass, Gateway, RouteUpstreamLimitsFilter, HTTPRoute)
`hack/run-upstream-limits-acceptance.sh`	Local kind cluster runner script

Unit tests

`TestEntriesEqual_APIGateway_Defaults` — `diff_test.go`

Covers apiGatewaysEqual with all combinations of the Defaults field:

Case	Expected
Both `nil` Defaults	Equal
Identical `MaxConnections`	Equal
`nil` vs non-nil Defaults	Not equal
Different `MaxConnections`	Not equal
Identical `PassiveHealthCheck`	Equal
Different `PassiveHealthCheck.MaxFailures`	Not equal
`nil` vs set `PassiveHealthCheck`	Not equal

`TestEntriesEqual_HTTPRoute_Limits` — `diff_test.go`

Covers httpServicesEqual with all combinations of the per-service Limits field:

Case	Expected
Both `nil` Limits	Equal
Identical `MaxConnections`	Equal
`nil` vs non-nil Limits	Not equal
Different `MaxConnections`	Not equal
Identical `PassiveHealthCheck`	Equal
Different `PassiveHealthCheck.MaxFailures`	Not equal
`nil` vs set `PassiveHealthCheck`	Not equal

Acceptance tests

Test: TestAPIGateway_UpstreamLimits_And_PassiveHealthCheck
File: acceptance/tests/api-gateway/api_gateway_upstream_limits_test.go
Run time: ~3.5 minutes (single Helm install amortised across all sub-tests)

All 6 sub-tests passed on a local Kind cluster:

--- PASS: TestAPIGateway_UpstreamLimits_And_PassiveHealthCheck (211.41s)
    --- PASS: .../day1/consul-config-entry-has-limits                       (0.00s)
    --- PASS: .../day1/envoy-config-has-circuit-breaker-and-outlier-detection  (2.14s)
    --- PASS: .../day2/update-limits-and-verify                             (2.13s)
    --- PASS: .../day1/gateway-defaults-via-annotations                     (2.14s)
    --- PASS: .../day2/update-gateway-defaults-and-verify                   (2.02s)
    --- PASS: .../service-level-overrides-gateway-defaults                  (4.17s)

Sub-test breakdown

`day1/consul-config-entry-has-limits`

Apply the RouteUpstreamLimitsFilter fixture (maxConnections=25, maxPendingRequests=50,
maxConcurrentRequests=100, passiveHealthCheck.interval=10s, maxFailures=3).
Assert that the Consul HTTPRouteConfigEntry has a non-nil Limits block on the backend
service with exactly those values.

`day1/envoy-config-has-circuit-breaker-and-outlier-detection`

Port-forward to the gateway pod's Envoy admin port (:19000) and fetch /config_dump.
Parse ClustersConfigDump.dynamic_active_clusters to find the static-server cluster.
Assert:

circuit_breakers.thresholds[0].max_connections == 25
outlier_detection block is present

Note: /clusters?format=json only returns runtime health stats, not the full xDS
circuit-breaker or outlier-detection config. /config_dump must be used.

`day2/update-limits-and-verify`

Patch the RouteUpstreamLimitsFilter in-place:

maxConnections 25 → 75, maxPendingRequests 50 → 150, maxConcurrentRequests 100 → 300
passiveHealthCheck.maxFailures 3 → 7

Assert:

Consul HTTPRouteConfigEntry reflects maxConnections=75, maxFailures=7
Envoy /config_dump shows max_connections: 75 and outlier_detection present

`day1/gateway-defaults-via-annotations`

Annotate the Gateway object:

api-gateway.consul.hashicorp.com/default-max-connections: "40"
api-gateway.consul.hashicorp.com/default-passive-health-check-interval: "15s"
api-gateway.consul.hashicorp.com/default-passive-health-check-max-failures: "5"
api-gateway.consul.hashicorp.com/default-passive-health-check-max-ejection-percent: "20"

Assert that the Consul APIGatewayConfigEntry.Defaults block is non-nil with
MaxConnections=40, PassiveHealthCheck.Interval=15s, MaxFailures=5.

`day2/update-gateway-defaults-and-verify`

Patch annotations: default-max-connections 40 → 80, default-passive-health-check-max-failures 5 → 9.
Assert Consul APIGatewayConfigEntry.Defaults.MaxConnections=80, MaxFailures=9.

`service-level-overrides-gateway-defaults`

With gateway-wide default at max_connections=40, update the RouteUpstreamLimitsFilter
to maxConnections=100. Assert:

Consul HTTPRouteConfigEntry carries Limits.MaxConnections=100 (not 40) — proving
the controller correctly uses service-level values over gateway defaults.
Envoy /config_dump shows circuit_breakers.max_connections: 100 — proving the
xDS push reflects the override end-to-end.

Local testing setup

Prerequisites

# Tools
kind v0.24+, kubectl, helm 3, go 1.22+

# Docker socket (colima / Docker Desktop on macOS)
export DOCKER_HOST="unix:///Users/<you>/.docker/run/docker.sock"

Build local images

# consul (from /path/to/consul repo)
cd /path/to/consul
make dev-docker                          # produces consul:local

# consul-k8s control-plane
cd /path/to/consul-k8s
make control-plane-dev-docker            # produces consul-k8s-control-plane:local

# consul-dataplane — pull arm64 digest directly (avoids broken multi-arch manifest)
docker pull hashicorp/consul-dataplane@sha256:4b116818a27d8032a136789c5cbd585a6e0560f6d21e7cacf77564799236d67e
docker tag  hashicorp/consul-dataplane@sha256:4b116818a27d8032a136789c5cbd585a6e0560f6d21e7cacf77564799236d67e \
            hashicorp/consul-dataplane:2.0-arm64
docker save hashicorp/consul-dataplane:2.0-arm64 -o /tmp/consul-dataplane-2.0-arm64-only.tar

Create Kind cluster and load images

kind create cluster --name dc1 \
  --image kindest/node:v1.30.2@sha256:ecfe5841b9bee4fe9690f49c118c33629fa345e3350a0c67a5a34482a99d6bba \
  --config acceptance/framework/environment/kind/kind.config

kind load docker-image --name dc1 consul:local
kind load docker-image --name dc1 consul-k8s-control-plane:local
kind load image-archive --name dc1 /tmp/consul-dataplane-2.0-arm64-only.tar

Run the acceptance test

kubectl config use-context kind-dc1

cd acceptance
go test ./tests/api-gateway/... -v \
  -run  "TestAPIGateway_UpstreamLimits_And_PassiveHealthCheck" \
  -timeout 60m -p 1 \
  -consul-image="consul:local" \
  -consul-k8s-image="consul-k8s-control-plane:local" \
  -consul-dataplane-image="hashicorp/consul-dataplane:2.0-arm64" \
  -use-kind -no-cleanup-on-failure

Or use the convenience script:

./hack/run-upstream-limits-acceptance.sh

Verify Envoy config manually

# Find the gateway pod
GW_POD=$(kubectl get pods -l gateway.consul.hashicorp.com/name=limits-gateway \
           -o jsonpath='{.items[0].metadata.name}')

# Port-forward to Envoy admin
kubectl port-forward $GW_POD 19000:19000 &

# Inspect the static-server cluster — circuit-breakers and outlier_detection
curl -s http://localhost:19000/config_dump | \
  python3 -c "
import json, sys
d = json.load(sys.stdin)
for section in d['configs']:
    if 'ClustersConfigDump' in section.get('@type',''):
        for c in section.get('dynamic_active_clusters', []):
            cl = c.get('cluster', {})
            if 'static-server' in cl.get('name',''):
                print(json.dumps(cl, indent=2))
"

Expected output (day-1):

{
  "name": "static-server.default.dc1.internal.<id>.consul",
  "circuit_breakers": {
    "thresholds": [{ "max_connections": 25, "max_pending_requests": 50, "max_requests": 100 }]
  },
  "outlier_detection": {
    "consecutive_5xx": 3,
    "interval": "10s",
    "base_ejection_time": "30s",
    "max_ejection_percent": 50,
    "enforcing_consecutive_5xx": 100
  }
}

…nd passive health checking

…ve health checks

…d PHC

…cks for API Gateway

…ive health checks

…ithub.com/hashicorp/consul-k8s into bharath/api-gateway-passive-health-checks

github-actions · 2026-08-14T10:39:32Z

Go Test Coverage: 61.5%

Patch coverage: 61.8% (136/220 changed lines covered)

See the workflow run for the full per-package breakdown and downloadable HTML report.

codecov-commenter · 2026-08-14T10:55:05Z

Codecov Report

❌ Patch coverage is 60.77348% with 71 lines in your changes missing coverage. Please review.
✅ Project coverage is 58.67%. Comparing base (cb007b6) to head (18b6128).
⚠️ Report is 12 commits behind head on main.

Files with missing lines	Patch %	Lines
...ontrol-plane/api/v1alpha1/zz_generated.deepcopy.go	0.00%	59 Missing ⚠️
...lane/api-gateway/controllers/gateway_controller.go	0.00%	8 Missing ⚠️
control-plane/api-gateway/common/translation.go	97.67%	1 Missing and 1 partial ⚠️
...ne/api/v1alpha1/routeupstreamlimitsfilter_types.go	50.00%	2 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #5596      +/-   ##
==========================================
+ Coverage   58.62%   58.67%   +0.04%     
==========================================
  Files         298      299       +1     
  Lines       36767    36945     +178     
==========================================
+ Hits        21554    21676     +122     
- Misses      13268    13326      +58     
+ Partials     1945     1943       -2

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

…s test

bharath-k1999 added 2 commits July 15, 2026 15:44

feat: add RouteUpstreamLimitsFilter for per-service upstream limits a…

2a78b69

…nd passive health checking

chore: bump go version to 1.26.5 in control-plane go.mod

b9bb06b

bharath-k1999 requested review from a team as code owners August 14, 2026 08:22

bharath-k1999 added 7 commits August 14, 2026 13:58

Merge branch 'main' into bharath/api-gateway-passive-health-checks

4a0e4be

chore: update consul/api replace directive to use phc-test version

91e98fb

test(api-gateway): add acceptance tests for upstream limits and passi…

1ace0be

…ve health checks

chore: add local acceptance test runner script for upstream limits an…

c5dd6b6

…d PHC

feat(#5596): add RouteUpstreamLimitsFilter CRD and passive health che…

1e0d749

…cks for API Gateway

chore: add changelog for PR 5596 api-gateway upstream limits and pass…

f70b345

…ive health checks

Merge branch 'bharath/api-gateway-passive-health-checks' of https://g…

c9c3307

…ithub.com/hashicorp/consul-k8s into bharath/api-gateway-passive-health-checks

test(api-gateway): remove unused helper functions from upstream limit…

18b6128

…s test

Navigation Menu

Uh oh!

feat: add RouteUpstreamLimitsFilter for per-service upstream limits and passive health checking - #5596

feat: add RouteUpstreamLimitsFilter for per-service upstream limits and passive health checking#5596
bharath-k1999 wants to merge 10 commits into
mainfrom
bharath/api-gateway-passive-health-checks

bharath-k1999 commented Aug 14, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Aug 14, 2026 •

edited

Loading

Uh oh!

codecov-commenter commented Aug 14, 2026 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

bharath-k1999 commented Aug 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

consul-k8s PR #5596 — API Gateway: Passive Health Checks & Upstream Limits

Summary

What is implemented

New CRD: RouteUpstreamLimitsFilter

Gateway-wide defaults via annotations

Files changed

Unit tests

TestEntriesEqual_APIGateway_Defaults — diff_test.go

TestEntriesEqual_HTTPRoute_Limits — diff_test.go

Acceptance tests

Sub-test breakdown

day1/consul-config-entry-has-limits

day1/envoy-config-has-circuit-breaker-and-outlier-detection

day2/update-limits-and-verify

day1/gateway-defaults-via-annotations

day2/update-gateway-defaults-and-verify

service-level-overrides-gateway-defaults

Local testing setup

Prerequisites

Build local images

Create Kind cluster and load images

Run the acceptance test

Verify Envoy config manually

Uh oh!

github-actions Bot commented Aug 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Go Test Coverage: 61.5%

Uh oh!

codecov-commenter commented Aug 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

bharath-k1999 commented Aug 14, 2026 •

edited

Loading

New CRD: `RouteUpstreamLimitsFilter`

`TestEntriesEqual_APIGateway_Defaults` — `diff_test.go`

`TestEntriesEqual_HTTPRoute_Limits` — `diff_test.go`

`day1/consul-config-entry-has-limits`

`day1/envoy-config-has-circuit-breaker-and-outlier-detection`

`day2/update-limits-and-verify`

`day1/gateway-defaults-via-annotations`

`day2/update-gateway-defaults-and-verify`

`service-level-overrides-gateway-defaults`

github-actions Bot commented Aug 14, 2026 •

edited

Loading

codecov-commenter commented Aug 14, 2026 •

edited

Loading