Forward MCPServerEntry headerForward to vMCP outbound requests

[ChrisJBurns]

Summary

Closes #4996.

MCPServerEntry.spec.headerForward.{addPlaintextHeaders,addHeadersFromSecret} was accepted by the CRD but never sent on outbound requests when the entry was consumed as a static backend of a VirtualMCPServer. Only MCPRemoteProxy forwarded the field — vMCP requests to remoteUrl arrived without the configured headers, breaking use cases like GitHub Copilot's X-MCP-Toolsets multi-toolset selection.

This PR fixes the bug without touching pkg/vmcp/config by mirroring MCPRemoteProxy's existing pattern: the operator emits per-(entry, header) env vars on the vMCP pod (literal values for plaintext, valueFrom.secretKeyRef for secrets), and the vMCP runtime walks the well-known prefixes at startup to reconstruct Backend.HeaderForward in static mode. TOOLHIVE_OTEL_HEADER_* already establishes plaintext-header-via-env in this codebase, so the convention isn't new.

Zero CRD/docs diff. Zero non-test changes under pkg/vmcp/config/.

Medium level

• Operator: buildHeaderForwardEnvVarsForEntries now emits literal-value env vars for addPlaintextHeaders alongside the existing valueFrom.secretKeyRef env vars for addHeadersFromSecret. Header normalization is extracted into a private normalizeHeaderForEnvVar helper that both GenerateHeaderForwardSecretEnvVarName and the new GenerateHeaderForwardPlaintextEnvVarName share, so the secret and plaintext branches can never diverge on a header that round-trips through one and not the other.
• Runtime: a new readHeaderForwardFromEnv (pkg/vmcp/cli/header_forward_env.go) walks os.Environ() for the TOOLHIVE_HEADER_PLAINTEXT_* and TOOLHIVE_SECRET_HEADER_FORWARD_* prefixes at startup, parses each (entry, header) pair via the inverse of the operator's normalization, and builds map[backendName]*vmcp.HeaderForwardConfig. Stray env vars whose decoded entry segment doesn't match a known static backend are dropped.
• Discoverer: NewUnifiedBackendDiscovererWithStaticBackends gains a headerForwardByBackend parameter. discoverFromStaticConfig attaches the matching map entry to Backend.HeaderForward by backend name.
• Round tripper (pkg/vmcp/client/header_forward.go): inserted between identityPropagatingRoundTripper (outer) and authRoundTripper (inner). Resolves plaintext + secret headers once at client-factory time via secrets.EnvironmentProvider. Rejects restricted headers via the shared pkg/transport/middleware.RestrictedHeaders set. Auth always wins over user-supplied headers because it runs after this tripper.
• Health monitor: BackendTarget construction in pkg/vmcp/health/monitor.go::performHealthCheck now carries HeaderForward, CABundlePath, and CABundleData so health probes hit backends with the same TLS trust and header injection as list/call traffic.
• MCPServerEntry reconciler: a new HeaderSecretRefsValidated condition (reusing MCPRemoteProxy's HeaderSecretNotFound reason) flips the entry to Failed when a referenced Secret is missing. GenerateHeaderForwardSecretEnvVarName now takes ownerName rather than proxyName so both MCPRemoteProxy and MCPServerEntry share one helper.
• Dynamic mode (K8s API discovery): pkg/vmcp/workloads/k8s.go::mcpServerEntryToBackend is unchanged — it reads headerForward directly from the MCPServerEntry CRD at backend-construction time, so no env-var path is needed there.

Low level

File	Change
pkg/vmcp/types.go	Add HeaderForwardConfig (no kubebuilder markers — runtime-only); add HeaderForward field on Backend and BackendTarget.
pkg/vmcp/registry.go	BackendToTarget copies HeaderForward.
pkg/vmcp/health/monitor.go	Carry HeaderForward, CABundlePath, CABundleData into the health-check BackendTarget.
pkg/vmcp/client/header_forward.go	New: headerForwardRoundTripper, buildHeaderForwardTripper, resolveHeaderForward.
pkg/vmcp/client/header_forward_test.go	Round-tripper, resolver, restricted-header rejection, end-to-end httptest.Server test.
pkg/vmcp/client/client.go	Insert tripper in chain (identity → headerForward → auth → http); add secretsProvider field.
pkg/vmcp/cli/header_forward_env.go	New: readHeaderForwardFromEnv plus header-name suffix splitter.
pkg/vmcp/cli/header_forward_env_test.go	Plaintext/secret/mixed/stray/multi-underscore-header table-driven tests.
pkg/vmcp/cli/serve.go	Build per-backend HeaderForward map from env in static-mode bootstrap; pass to discoverer.
pkg/vmcp/aggregator/discoverer.go	New headerForwardByBackend field on backendDiscoverer; constructor parameter; lookup in discoverFromStaticConfig.
pkg/vmcp/aggregator/discoverer_test.go	Pass nil for new parameter at four existing call sites.
cmd/thv-operator/api/v1beta1/mcpserverentry_types.go	HeaderSecretRefsValidated condition + reasons.
cmd/thv-operator/controllers/mcpserverentry_controller.go	validateHeaderForwardSecretRefs wired into reconcile.
cmd/thv-operator/controllers/virtualmcpserver_deployment.go	Extend buildHeaderForwardEnvVarsForEntries to emit plaintext env vars in sorted order alongside secret refs.
cmd/thv-operator/pkg/controllerutil/externalauth.go	Extract normalizeHeaderForEnvVar; add GenerateHeaderForwardPlaintextEnvVarName; rename proxyName → ownerName so MCPRemoteProxy and MCPServerEntry share one helper.

Type of change

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change
• Refactoring
• Documentation
• Other

Test plan

• task build passes
• task test passes for all touched packages
• New unit tests:
  • TestBuildHeaderForwardEnvVarsForEntries — operator-side plaintext + secret + mixed + sort-determinism
  • TestReadHeaderForwardFromEnv — runtime-side env walk, stray-var drop, multi-underscore header splitting
  • TestHeaderForwardRoundTripper_*, TestResolveHeaderForward_*, TestBuildHeaderForwardTripper_*, end-to-end httptest.Server test (cherry-picked from PR Forward MCPServerEntry headerForward to vMCP outbound requests #5013)
  • MCPServerEntry reconciler HeaderSecretsValid / HeaderSecretNotFound cases (cherry-picked from PR Forward MCPServerEntry headerForward to vMCP outbound requests #5013)
• task operator-manifests, task operator-generate, task crdref-gen produce zero diff under deploy/charts/operator-crds/ and docs/operator/crd-api.md
• Zero non-test changes under pkg/vmcp/config/

Special notes for reviewers

• Why not the wrapper from DON'T REVIEW YET: Introduce RuntimeConfig wrapper for vMCP ConfigMap surface #5238? That PR set up runtime.Config so future operator-resolved sidecar fields could ship via the ConfigMap without leaking into the CRD. After investigation, env vars are a better fit for headerForward specifically — MCPRemoteProxy already uses env vars for the single-backend version of the same problem, and TOOLHIVE_OTEL_HEADER_* already establishes plaintext-header-via-env in this codebase. The wrapper stays empty for a future field that genuinely benefits from YAML co-location with user-authored vMCP config.
• Plaintext-via-env exposure: literal header values land in kubectl describe pod output instead of kubectl get configmap. Both are RBAC-gated under similar verbs. Truly sensitive values still ride valueFrom.secretKeyRef and never enter the operator's view of the world. Plaintext header values are by definition non-secret — that's why the user chose plaintext.
• Header name normalization is one-way: the runtime sees normalized header names (uppercased, hyphens to underscores). HTTP header matching is canonical-case-insensitive so this is safe for the round tripper's http.Header.Set (which canonicalizes regardless).
• Implementation plan: an approved design doc was produced before this PR (committed as DESIGN.md during development, removed before final commit since .claude/rules/pr-creation.md keeps planning artifacts out of the PR diff).

Manual test walkthroughs

The two collapsible blocks below contain self-contained step-by-step guides
for verifying MCPServerEntry.headerForward end-to-end against a kind
cluster — one per backend-discovery mode. Each guide stands alone:
prerequisites, setup, traffic generation, header-capture verification, and
cleanup. Identical work was used to validate this PR locally.

Static mode — outgoingAuth.source: inline

This walkthrough verifies that operator-emitted headerForward env vars are
parsed by vMCP at startup and that both plaintext and Secret-backed
headers are injected on every outbound MCP request.

Static mode is selected by VirtualMCPServer.spec.outgoingAuth.source: inline.
The operator pre-renders backends into the vMCP ConfigMap and emits
TOOLHIVE_HEADER_FORWARD_* env vars; vMCP reads them at startup.

---

0. Prerequisites

• kind cluster running with the ToolHive operator built from PR Forward MCPServerEntry headerForward to vMCP outbound requests #5239 (or cburns/headerforward-envvar).
  • From the PR worktree: task kind-with-toolhive-operator-local (first time)
  • Or task operator-deploy-local if the cluster is already up.
• kubectl pointing at the cluster (KUBECONFIG=./kconfig.yaml).
• Working directory: the PR worktree
  (.claude/worktrees/headerforward-envvar if you used Agent's isolated
  worktree, otherwise wherever you checked the branch out).

All commands below assume:

cd /path/to/toolhive               # branch: cburns/headerforward-envvar
export KUBECONFIG=./kconfig.yaml   # or your kind cluster kubeconfig

---

1. Create the test namespace and Secret

kubectl create namespace hf-test --dry-run=client -o yaml | kubectl apply -f -

kubectl -n hf-test create secret generic test-secret \
  --from-literal=token=secret-from-k8s-secret \
  --dry-run=client -o yaml | kubectl apply -f -

The Secret value secret-from-k8s-secret is what we expect to see arrive at
the backend in the X-Api-Key header.

---

2. Deploy the header-capturing echo backend

This is a tiny Python HTTP server that:

• Logs every incoming request's headers
• Returns a minimal MCP initialize response so vMCP doesn't error
• Exposes GET /headers returning all captured headers as JSON

cat <<'EOF' | kubectl apply -n hf-test -f -
apiVersion: v1
kind: ConfigMap
metadata:
  name: echo-mcp-script
data:
  server.py: |
    import json, threading
    from http.server import BaseHTTPRequestHandler, HTTPServer
    captured_headers = []
    captured_lock = threading.Lock()
    class H(BaseHTTPRequestHandler):
        def _log_headers(self):
            with captured_lock:
                captured_headers.append({k: v for k, v in self.headers.items()})
        def do_GET(self):
            if self.path == "/headers":
                with captured_lock:
                    body = json.dumps(captured_headers).encode()
                self.send_response(200)
                self.send_header("Content-Type", "application/json")
                self.send_header("Content-Length", str(len(body)))
                self.end_headers()
                self.wfile.write(body)
                return
            self._log_headers()
            self.send_response(200)
            self.send_header("Content-Type", "application/json")
            self.end_headers()
            self.wfile.write(b'{"jsonrpc":"2.0","id":0,"result":{}}')
        def do_POST(self):
            self._log_headers()
            length = int(self.headers.get("Content-Length", "0"))
            _ = self.rfile.read(length)
            body = b'{"jsonrpc":"2.0","id":1,"result":{"protocolVersion":"2025-03-26","capabilities":{"tools":{}},"serverInfo":{"name":"echo","version":"0"}}}'
            self.send_response(200)
            self.send_header("Content-Type", "application/json")
            self.send_header("Mcp-Session-Id", "fake-session-1")
            self.send_header("Content-Length", str(len(body)))
            self.end_headers()
            self.wfile.write(body)
        def log_message(self, *_): pass
    HTTPServer(("0.0.0.0", 8080), H).serve_forever()
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: echo-mcp
spec:
  replicas: 1
  selector: { matchLabels: { app: echo-mcp } }
  template:
    metadata: { labels: { app: echo-mcp } }
    spec:
      containers:
        - name: echo
          image: python:3.13-slim
          command: ["python3", "/scripts/server.py"]
          ports: [{ containerPort: 8080 }]
          volumeMounts:
            - { name: script, mountPath: /scripts }
      volumes:
        - name: script
          configMap: { name: echo-mcp-script }
---
apiVersion: v1
kind: Service
metadata:
  name: echo-mcp
spec:
  selector: { app: echo-mcp }
  ports: [{ port: 80, targetPort: 8080 }]
EOF

kubectl -n hf-test rollout status deploy/echo-mcp --timeout=60s

---

3. Make the backend reachable from a non-blocked hostname

The operator validates MCPServerEntry.spec.remoteUrl against a
SSRF blocklist that rejects cluster.local, RFC1918 ranges, loopback, etc.
For testing we bind a fake public hostname (echo-public.test) to echo-mcp's
ClusterIP via CoreDNS.

ECHO_IP=$(kubectl -n hf-test get svc echo-mcp -o jsonpath='{.spec.clusterIP}')
echo "echo-mcp ClusterIP: ${ECHO_IP}"

# Patch CoreDNS to add a hosts block for echo-public.test → ${ECHO_IP}
kubectl -n kube-system get configmap coredns -o json \
  | jq --arg ip "${ECHO_IP}" '
      .data.Corefile |= (
        if test("echo-public.test") then .
        else . + "\n    echo-public.test:53 {\n        hosts {\n            \($ip) echo-public.test\n            fallthrough\n        }\n    }\n"
        end
      )
    ' \
  | kubectl apply -f -

kubectl -n kube-system rollout restart deploy/coredns
kubectl -n kube-system rollout status deploy/coredns --timeout=60s

Verify:

kubectl -n hf-test run dnscheck --rm -it --restart=Never \
  --image=busybox -- nslookup echo-public.test
# Should resolve to the ClusterIP from above.

---

4. Apply the static-mode VirtualMCPServer

cat <<'EOF' | kubectl apply -f -
---
apiVersion: toolhive.stacklok.dev/v1beta1
kind: MCPGroup
metadata: { name: hf-group, namespace: hf-test }
spec: {}
---
apiVersion: toolhive.stacklok.dev/v1beta1
kind: MCPServerEntry
metadata:
  name: github-copilot-fake
  namespace: hf-test
spec:
  remoteUrl: http://echo-public.test/
  transport: streamable-http
  groupRef: { name: hf-group }
  headerForward:
    addPlaintextHeaders:
      X-MCP-Toolsets: "projects,issues,pull_requests"
      X-Trace-Id: "kind-test-static"
    addHeadersFromSecret:
      - headerName: X-Api-Key
        valueSecretRef:
          name: test-secret
          key: token
---
apiVersion: toolhive.stacklok.dev/v1beta1
kind: VirtualMCPServer
metadata:
  name: vmcp-headerfwd
  namespace: hf-test
spec:
  config:
    aggregation:
      conflictResolution: prefix
      conflictResolutionConfig:
        prefixFormat: '{workload}_'
  groupRef: { name: hf-group }
  incomingAuth: { type: anonymous }
  outgoingAuth: { source: inline }   # <-- STATIC MODE
EOF

Wait until everything is Ready:

kubectl -n hf-test wait --for=condition=Ready --timeout=120s \
  virtualmcpserver/vmcp-headerfwd
kubectl -n hf-test get mcpserverentry github-copilot-fake \
  -o jsonpath='{.status.phase}{"\n"}'   # expect: Valid

---

5. Verify the operator emitted the right env vars

This is the operator-side acceptance check: commit "Emit headerForward env vars
from VirtualMCPServer deployment" should produce:

• A TOOLHIVE_HEADER_FORWARD_<entry> env var with the JSON manifest
• A TOOLHIVE_SECRET_HEADER_FORWARD_<header>_<entry> env var sourced from the Secret

kubectl -n hf-test describe deploy vmcp-headerfwd | grep -E 'TOOLHIVE_HEADER_FORWARD|TOOLHIVE_SECRET_HEADER_FORWARD'

You should see something like:

TOOLHIVE_HEADER_FORWARD_GITHUB_COPILOT_FAKE: {"addPlaintextHeaders":{"X-MCP-Toolsets":"projects,issues,pull_requests","X-Trace-Id":"kind-test-static"},"addHeadersFromSecret":{"X-Api-Key":"HEADER_FORWARD_X_API_KEY_GITHUB_COPILOT_FAKE"}}
TOOLHIVE_SECRET_HEADER_FORWARD_X_API_KEY_GITHUB_COPILOT_FAKE: <set to the key 'token' in secret 'test-secret'>

---

6. Verify vMCP started in static mode

kubectl -n hf-test logs deploy/vmcp-headerfwd | grep -E 'Static mode|loaded static backend' | head -3

Expected:

Static mode: using 1 pre-configured backends
loaded static backend  name=github-copilot-fake  url=http://echo-public.test/  transport=streamable-http

---

7. Reset the backend's capture, then trigger MCP traffic

# Restart echo-mcp so its in-process capture list starts empty
kubectl -n hf-test rollout restart deploy/echo-mcp
kubectl -n hf-test rollout status deploy/echo-mcp --timeout=60s

# Port-forward both vMCP and the echo backend
kubectl -n hf-test port-forward svc/vmcp-vmcp-headerfwd 4483:4483 \
  >/tmp/vmcp-pf.log 2>&1 &
kubectl -n hf-test port-forward svc/echo-mcp 8080:80 \
  >/tmp/echo-pf.log 2>&1 &
sleep 5

# Send a few initialize requests through vMCP
for i in 1 2 3; do
  curl -s -X POST http://localhost:4483/mcp \
    -H "Content-Type: application/json" \
    -H "Accept: application/json, text/event-stream" \
    -d "{\"jsonrpc\":\"2.0\",\"method\":\"initialize\",\"params\":{\"protocolVersion\":\"2025-03-26\",\"capabilities\":{},\"clientInfo\":{\"name\":\"t-${i}\",\"version\":\"1\"}},\"id\":${i}}" \
    >/dev/null
  sleep 1
done

# Wait at least 35s for a health-check probe
# (default health interval is 30s; health-check requests use the per-backend client too)
sleep 35

---

8. Verify the captured headers

curl -s http://localhost:8080/headers | python3 -c "
import json, sys
data = json.loads(sys.stdin.read())
print(f'Total backend requests captured: {len(data)}')
forwarded = [d for d in data if any(k in d for k in ('X-Api-Key','X-Trace-Id','X-Mcp-Toolsets'))]
print(f'Requests with forwarded headers:  {len(forwarded)}')
if forwarded:
    print('--- Example forwarded request headers ---')
    ex = {k: v for k, v in forwarded[0].items() if k.startswith(('X-','x-'))}
    print(json.dumps(ex, indent=2))
"

Pass criteria:

• At least one captured request contains all three headers:
  • X-Trace-Id: kind-test-static (plaintext)
  • X-MCP-Toolsets: projects,issues,pull_requests (plaintext)
  • X-Api-Key: secret-from-k8s-secret (resolved from test-secret/token)

Not every captured request will carry the headers. The first 2–3 captures
are the discovery/initialize handshake which uses an internal HTTP path that
bypasses the per-backend decorator. Once routing is established, the
initialize-via-decorator and the periodic health-check probes carry the
headers — that is what proves the feature works.

---

9. Cleanup

pkill -f "kubectl port-forward" 2>/dev/null

kubectl delete namespace hf-test

# Revert the CoreDNS rewrite (optional, only if you don't want to keep the test setup)
kubectl -n kube-system get configmap coredns -o json \
  | jq '.data.Corefile |= sub("\\s*echo-public\\.test:53 \\{[^}]*hosts \\{[^}]*\\}[^}]*\\}\\s*"; "")' \
  | kubectl apply -f -
kubectl -n kube-system rollout restart deploy/coredns

Dynamic mode — outgoingAuth.source: discovered

This walkthrough verifies that vMCP's K8s workload discoverer reads
MCPServerEntry.spec.headerForward directly from the API server at runtime
and that both plaintext and Secret-backed headers are injected on
every outbound MCP request.

Dynamic mode is selected by VirtualMCPServer.spec.outgoingAuth.source: discovered.
vMCP runs a controller-runtime watcher that reconciles backends into a
DynamicRegistry; pkg/vmcp/workloads/k8s.go::headerForwardFromEntry projects
spec.headerForward onto each vmcp.Backend.

---

0. Prerequisites

• kind cluster running with the ToolHive operator built from PR Forward MCPServerEntry headerForward to vMCP outbound requests #5239 (or cburns/headerforward-envvar).
  • From the PR worktree: task kind-with-toolhive-operator-local (first time)
  • Or task operator-deploy-local if the cluster is already up.
• kubectl pointing at the cluster (KUBECONFIG=./kconfig.yaml).
• Working directory: the PR worktree (.claude/worktrees/headerforward-envvar).

All commands below assume:

cd /path/to/toolhive               # branch: cburns/headerforward-envvar
export KUBECONFIG=./kconfig.yaml

---

1. Create the test namespace and Secret

kubectl create namespace hf-test --dry-run=client -o yaml | kubectl apply -f -

kubectl -n hf-test create secret generic test-secret \
  --from-literal=token=secret-from-k8s-secret \
  --dry-run=client -o yaml | kubectl apply -f -

The Secret value secret-from-k8s-secret is what we expect to see arrive at
the backend in the X-Api-Key header.

---

2. Deploy the header-capturing echo backend

This is a tiny Python HTTP server that:

• Logs every incoming request's headers
• Returns a minimal MCP initialize response so vMCP doesn't error
• Exposes GET /headers returning all captured headers as JSON

cat <<'EOF' | kubectl apply -n hf-test -f -
apiVersion: v1
kind: ConfigMap
metadata:
  name: echo-mcp-script
data:
  server.py: |
    import json, threading
    from http.server import BaseHTTPRequestHandler, HTTPServer
    captured_headers = []
    captured_lock = threading.Lock()
    class H(BaseHTTPRequestHandler):
        def _log_headers(self):
            with captured_lock:
                captured_headers.append({k: v for k, v in self.headers.items()})
        def do_GET(self):
            if self.path == "/headers":
                with captured_lock:
                    body = json.dumps(captured_headers).encode()
                self.send_response(200)
                self.send_header("Content-Type", "application/json")
                self.send_header("Content-Length", str(len(body)))
                self.end_headers()
                self.wfile.write(body)
                return
            self._log_headers()
            self.send_response(200)
            self.send_header("Content-Type", "application/json")
            self.end_headers()
            self.wfile.write(b'{"jsonrpc":"2.0","id":0,"result":{}}')
        def do_POST(self):
            self._log_headers()
            length = int(self.headers.get("Content-Length", "0"))
            _ = self.rfile.read(length)
            body = b'{"jsonrpc":"2.0","id":1,"result":{"protocolVersion":"2025-03-26","capabilities":{"tools":{}},"serverInfo":{"name":"echo","version":"0"}}}'
            self.send_response(200)
            self.send_header("Content-Type", "application/json")
            self.send_header("Mcp-Session-Id", "fake-session-1")
            self.send_header("Content-Length", str(len(body)))
            self.end_headers()
            self.wfile.write(body)
        def log_message(self, *_): pass
    HTTPServer(("0.0.0.0", 8080), H).serve_forever()
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: echo-mcp
spec:
  replicas: 1
  selector: { matchLabels: { app: echo-mcp } }
  template:
    metadata: { labels: { app: echo-mcp } }
    spec:
      containers:
        - name: echo
          image: python:3.13-slim
          command: ["python3", "/scripts/server.py"]
          ports: [{ containerPort: 8080 }]
          volumeMounts:
            - { name: script, mountPath: /scripts }
      volumes:
        - name: script
          configMap: { name: echo-mcp-script }
---
apiVersion: v1
kind: Service
metadata:
  name: echo-mcp
spec:
  selector: { app: echo-mcp }
  ports: [{ port: 80, targetPort: 8080 }]
EOF

kubectl -n hf-test rollout status deploy/echo-mcp --timeout=60s

---

3. Make the backend reachable from a non-blocked hostname

The operator validates MCPServerEntry.spec.remoteUrl against a SSRF
blocklist that rejects cluster.local, RFC1918 ranges, loopback, etc. For
testing we bind a fake public hostname (echo-public.test) to echo-mcp's
ClusterIP via CoreDNS.

ECHO_IP=$(kubectl -n hf-test get svc echo-mcp -o jsonpath='{.spec.clusterIP}')
echo "echo-mcp ClusterIP: ${ECHO_IP}"

kubectl -n kube-system get configmap coredns -o json \
  | jq --arg ip "${ECHO_IP}" '
      .data.Corefile |= (
        if test("echo-public.test") then .
        else . + "\n    echo-public.test:53 {\n        hosts {\n            \($ip) echo-public.test\n            fallthrough\n        }\n    }\n"
        end
      )
    ' \
  | kubectl apply -f -

kubectl -n kube-system rollout restart deploy/coredns
kubectl -n kube-system rollout status deploy/coredns --timeout=60s

Verify:

kubectl -n hf-test run dnscheck --rm -it --restart=Never \
  --image=busybox -- nslookup echo-public.test

---

4. Apply the dynamic-mode VirtualMCPServer

The key differences from static mode:

• outgoingAuth.source: discovered instead of inline
• vMCP needs RBAC to watch CRDs (the operator creates this automatically when it sees source: discovered)

cat <<'EOF' | kubectl apply -f -
---
apiVersion: toolhive.stacklok.dev/v1beta1
kind: MCPGroup
metadata: { name: hf-group-dyn, namespace: hf-test }
spec: {}
---
apiVersion: toolhive.stacklok.dev/v1beta1
kind: MCPServerEntry
metadata:
  name: github-copilot-fake-dyn
  namespace: hf-test
spec:
  remoteUrl: http://echo-public.test/
  transport: streamable-http
  groupRef: { name: hf-group-dyn }
  headerForward:
    addPlaintextHeaders:
      X-MCP-Toolsets: "projects,issues,pull_requests"
      X-Trace-Id: "kind-test-dynamic"
    addHeadersFromSecret:
      - headerName: X-Api-Key
        valueSecretRef:
          name: test-secret
          key: token
---
apiVersion: toolhive.stacklok.dev/v1beta1
kind: VirtualMCPServer
metadata:
  name: vmcp-headerfwd-dyn
  namespace: hf-test
spec:
  config:
    aggregation:
      conflictResolution: prefix
      conflictResolutionConfig:
        prefixFormat: '{workload}_'
  groupRef: { name: hf-group-dyn }
  incomingAuth: { type: anonymous }
  outgoingAuth: { source: discovered }   # <-- DYNAMIC MODE
EOF

Wait until everything is Ready:

kubectl -n hf-test wait --for=condition=Ready --timeout=120s \
  virtualmcpserver/vmcp-headerfwd-dyn
kubectl -n hf-test get mcpserverentry github-copilot-fake-dyn \
  -o jsonpath='{.status.phase}{"\n"}'    # expect: Valid

---

5. Verify vMCP started in dynamic mode

kubectl -n hf-test logs deploy/vmcp-headerfwd-dyn \
  | grep -E 'dynamic mode|backend watcher|Successfully reconciled' | head -10

Expected (excerpt):

dynamic mode: initializing group manager for backend discovery
dynamic mode: discovering backends from K8s API
discovered backends in group  count=1  group=hf-group-dyn
detected dynamic backend discovery mode (outgoingAuth.source: discovered)
kubernetes backend watcher started for dynamic backend discovery
Successfully reconciled backend  controller=backend-reconciler-hf-group-dyn  name=github-copilot-fake-dyn  registryVersion=1

The "Successfully reconciled backend" line proves the K8s controller-runtime
watcher reconciled the MCPServerEntry into the DynamicRegistry. The
headerForward field on the entry is projected onto the runtime
vmcp.Backend at this step.

---

6. Reset the backend's capture, then trigger MCP traffic

# Restart echo-mcp so its in-process capture list starts empty
kubectl -n hf-test rollout restart deploy/echo-mcp
kubectl -n hf-test rollout status deploy/echo-mcp --timeout=60s

# Port-forward both vMCP and the echo backend
kubectl -n hf-test port-forward svc/vmcp-vmcp-headerfwd-dyn 4484:4483 \
  >/tmp/vmcp-dyn-pf.log 2>&1 &
kubectl -n hf-test port-forward svc/echo-mcp 8080:80 \
  >/tmp/echo-pf.log 2>&1 &
sleep 5

# Send a few initialize requests through vMCP
for i in 1 2 3; do
  curl -s -X POST http://localhost:4484/mcp \
    -H "Content-Type: application/json" \
    -H "Accept: application/json, text/event-stream" \
    -d "{\"jsonrpc\":\"2.0\",\"method\":\"initialize\",\"params\":{\"protocolVersion\":\"2025-03-26\",\"capabilities\":{},\"clientInfo\":{\"name\":\"t-${i}\",\"version\":\"1\"}},\"id\":${i}}" \
    >/dev/null
  sleep 1
done

# Wait at least 35s for a health-check probe
sleep 35

---

7. Verify the captured headers

curl -s http://localhost:8080/headers | python3 -c "
import json, sys
data = json.loads(sys.stdin.read())
print(f'Total backend requests captured: {len(data)}')
forwarded = [d for d in data if any(k in d for k in ('X-Api-Key','X-Trace-Id','X-Mcp-Toolsets'))]
print(f'Requests with forwarded headers:  {len(forwarded)}')
if forwarded:
    print('--- Example forwarded request headers ---')
    ex = {k: v for k, v in forwarded[0].items() if k.startswith(('X-','x-'))}
    print(json.dumps(ex, indent=2))
"

Pass criteria:

• At least one captured request contains all three headers:
  • X-Trace-Id: kind-test-dynamic (plaintext) — note the -dynamic suffix
    proves it came from this entry, not the static one if you happen to be
    running both side-by-side.
  • X-MCP-Toolsets: projects,issues,pull_requests (plaintext)
  • X-Api-Key: secret-from-k8s-secret (resolved from test-secret/token)

Not every captured request will carry the headers. The first 2–3 captures
are the discovery/initialize handshake which uses an internal HTTP path that
bypasses the per-backend decorator. Once routing is established, the
regular MCP traffic and the periodic health-check probes carry the headers.

---

8. Bonus: verify runtime updates flow through

Dynamic mode's payoff is that headerForward changes are picked up without
restarting vMCP. To prove it:

# Change the trace id at runtime (no pod restart)
kubectl -n hf-test patch mcpserverentry github-copilot-fake-dyn --type merge -p \
  '{"spec":{"headerForward":{"addPlaintextHeaders":{"X-Trace-Id":"updated-at-runtime","X-MCP-Toolsets":"projects,issues,pull_requests"}}}}'

# Reset captures and wait for the next health-check probe
kubectl -n hf-test rollout restart deploy/echo-mcp
kubectl -n hf-test rollout status deploy/echo-mcp --timeout=60s

# Re-establish port-forward to echo (it changes after restart)
pkill -f 'svc/echo-mcp 8080:80' 2>/dev/null
kubectl -n hf-test port-forward svc/echo-mcp 8080:80 \
  >/tmp/echo-pf.log 2>&1 &
sleep 35

curl -s http://localhost:8080/headers | python3 -c "
import json, sys
data = json.loads(sys.stdin.read())
trace_ids = {d.get('X-Trace-Id') for d in data if d.get('X-Trace-Id')}
print('X-Trace-Id values seen:', trace_ids)
"

You should see updated-at-runtime in the values — proving the K8s watcher
re-reconciled the entry, the DynamicRegistry was updated, and the next
outbound request used the new value. (No pod restart was performed.)

---

9. Cleanup

pkill -f "kubectl port-forward" 2>/dev/null

kubectl delete namespace hf-test

# Revert the CoreDNS rewrite (optional)
kubectl -n kube-system get configmap coredns -o json \
  | jq '.data.Corefile |= sub("\\s*echo-public\\.test:53 \\{[^}]*hosts \\{[^}]*\\}[^}]*\\}\\s*"; "")' \
  | kubectl apply -f -
kubectl -n kube-system rollout restart deploy/coredns

Large PR Justification

• I have separated what would be 7/8 PRs over 7/8 commits. Just to avoid a release happening and the code getting shipped half baked.

Generated with Claude Code

[github-actions]

Large PR Detected

This PR exceeds 1000 lines of changes and requires justification before it can be reviewed.

How to unblock this PR:

Add a section to your PR description with the following format:

## Large PR Justification

[Explain why this PR must be large, such as:]
- Generated code that cannot be split
- Large refactoring that must be atomic
- Multiple related changes that would break if separated
- Migration or data transformation

Alternative:

Consider splitting this PR into smaller, focused changes (< 1000 lines each) for easier review and reduced risk.

See our Contributing Guidelines for more details.

---

This review will be automatically dismissed once you add the justification section.

[codecov]

Codecov Report

❌ Patch coverage is 70.06803% with 88 lines in your changes missing coverage. Please review.
✅ Project coverage is 67.95%. Comparing base (6733a54) to head (c328915).

Files with missing lines	Patch %	Lines
...-operator/controllers/mcpserverentry_controller.go	43.61%	51 Missing and 2 partials ⚠️
...perator/controllers/virtualmcpserver_deployment.go	73.91%	9 Missing and 9 partials ⚠️
pkg/vmcp/client/header_forward.go	76.92%	9 Missing and 6 partials ⚠️
pkg/vmcp/client/client.go	60.00%	1 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #5239      +/-   ##
==========================================
- Coverage   67.97%   67.95%   -0.02%     
==========================================
  Files         612      615       +3     
  Lines       62723    63001     +278     
==========================================
+ Hits        42633    42810     +177     
- Misses      16908    16986      +78     
- Partials     3182     3205      +23     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Large PR justification has been provided. Thank you!

[github-actions]

✅ Large PR justification has been provided. The size review has been dismissed and this PR can now proceed with normal review.