Document vMCP scalability limits and operational configuration

[yrobla]

Summary

Several vMCP scalability limits and operational configuration details are either undocumented or only present in source code. This issue tracks what needs to be added to the docs, primarily in docs/toolhive/guides-vmcp/scaling-and-performance.mdx.

Gaps to address

1. Session cache capacity (1000 sessions/pod)

• What: Each vMCP pod holds a node-local LRU cache of 1000 live sessions. When full, least-recently-used sessions are evicted and their backend connections are closed abruptly.
• Where in code: pkg/vmcp/server/sessionmanager/factory.go — defaultCacheCapacity = 1000
• What to document: The limit, what happens on eviction, when operators might hit it, and guidance on sizing pod count accordingly.
• Note: This is not currently exposed as a CRD field — worth considering whether it should be.

2. Session TTL default (30 minutes)

• What: Sessions inactive for 30 minutes are automatically evicted from Redis and the local cache. The field exists in OperationalConfig in the CRD but the default is not surfaced in user-facing docs.
• Where in code: pkg/vmcp/server/server.go — defaultSessionTTL = 30 * time.Minute
• What to document: Default value, what happens when it expires (client must reconnect), and guidance on tuning for long-lived vs. short-lived workloads.

3. File descriptor limits

• What: Each session holds approximately one HTTP connection per backend. At 1000 sessions × 10 backends, a pod can hold ~10K file descriptors. This is not documented or accounted for in any container resource guidance.
• What to document: Expected FD consumption formula, recommended ulimit / Kubernetes container limits, and what happens if FDs are exhausted.

4. Cross-pod session rebuild cost

• What: When a request lands on a pod that doesn't have the session cached (cache miss or no sticky routing), RestoreSession reconnects to all backends and rebuilds the routing table. This costs ~100–500ms.
• What to document: This cost should be called out in the sticky session / Redis section so operators understand the latency tradeoff of not using session affinity, not just the correctness concern.

Related files

• docs/toolhive/guides-vmcp/scaling-and-performance.mdx — primary target
• docs/toolhive/reference/crd-spec.md — may need updates for TTL default and cache capacity

[danbarr]

Most of this was addressed by #799, which added a Capacity limits section to docs/toolhive/guides-vmcp/scaling-and-performance.mdx:

• Item 1 (1,000-session per-pod cache): covered - LRU cap, eviction behavior, Redis-backed transparent rebuild vs. lossy non-Redis case, and replicas × 1,000 scaling guidance.
• Item 2 (30-minute session TTL): covered - default, reinitialize-on-expiry behavior, and Redis sliding-window refresh. The CRD reference page doesn't surface the 30-minute default, but that page is auto-generated from the upstream CRD.
• Item 3 (file descriptor limits): covered - concurrent_sessions × backends_per_session formula, default 1,024 soft nofile, and guidance to raise it.

Remaining gap - Item 4 (cross-pod session rebuild cost): the rebuild mechanic is mentioned ("the session manager transparently rebuilds the session from stored metadata and reconnects to backends") but the ~100-500ms latency cost is not called out. The original ask was to document this in the sticky session / Redis section so operators understand the latency tradeoff of skipping session affinity, not just the correctness concern. Worth a small follow-up - either a sentence in the rebuild paragraph or in the ClientIP affinity warning quantifying the cost.

Up to you whether to keep this open for that follow-up or close and file a smaller issue for item 4.