JVM-aware vertical right-sizing for Kubernetes. Warden makes a running JVM give memory back to the OS during predictable low-traffic windows, then safely lowers the Pod's resource footprint — reclaiming idle infrastructure without killing the pod, dropping its warm state, or risking your high-availability posture.
In ~95% of production environments, Horizontal Pod Autoscaling (HPA) is the right answer: scale from 10 replicas to 2 when traffic drops, and you save money. Warden does not compete with that.
But there are three well-known scenarios where removing replicas is not an option, and shrinking the pods you keep is the only lever left:
High-availability architectures often can't scale below a floor — e.g. 3 sub-services × 3 replicas across 3 Availability Zones. If each JVM needs 4 GB at peak, that's 36 GB reserved overnight to serve zero requests. HPA can't touch it; the floor is there for failover, not for load.
Warden keeps all 9 replicas alive for availability but shrinks each one's footprint during the off-peak window — turning a 36 GB idle reservation into ~9 GB, without reducing replica count.
Many enterprise Java apps (older Spring Boot, WebSphere, large multi-module
builds) take 2–5 minutes to boot and pass readiness. If HPA scales you down to 2
pods overnight and a sharp spike hits at 06:00, the new pods sit in
ContainerCreating/unready for minutes while the surviving 2 get slammed, spike
CPU, and cascade into an outage.
Warden keeps the pods warm — JVM up, classes loaded, connection pools primed. Raising the footprint back to peak is a sub-second resize, not a cold start.
For an in-memory data grid, a stateful worker queue, or a distributed cache (this is mnemo-cache's home turf), you can't just kill a replica to save money — termination forces data rebalancing, purges local cache, and thrashes surviving nodes with replication overhead.
Warden instead shrinks in place: it flushes expired/evictable entries and idle pages, runs a deep GC, uncommits the freed pages back to the OS, and keeps the node running at a lower price point — with its hot working set intact.
A few load-bearing truths that shape the entire design:
Cost lives in requests, not limits. Lowering a Pod's memory limit
saves nothing — you still pay for the node. Kubernetes cost is driven by
requests, because that's what the scheduler bin-packs against and what
Cluster
Autoscaler
uses to decide it can drain and delete a node.
Warden's action boundary ends at "lower the request." It resizes an existing pod in place — it never changes the number of pods (that's HPA) or the number of nodes (that's the autoscaler). Everything past the request change is emergent cluster behaviour that Warden enables but does not perform, and does not guarantee:
The ⇢ is a hand-off, not a step Warden takes. On a fixed node pool with no
autoscaler, Warden's lower-request still frees schedulable capacity but zero
billed capacity — so the savings story is only as real as your consolidation
setup.
You can't change -Xmx on a running JVM. Max heap is fixed at launch. What
is movable at runtime is the soft ceiling and committed memory:
- ZGC —
SoftMaxHeapSize, settable live viajcmd VM.set_flag, plus automatic uncommit of unused heap. - Shenandoah — aggressive uncommit via
-XX:ShenandoahUncommitDelay(the gold standard for returning memory to the OS). - G1 —
-XX:G1PeriodicGCInterval/-XX:G1PeriodicGCSystemLoadThresholdalready trigger idle GC + uncommit.
The JVM already does half of this. Warden's job is the orchestration — it drives these knobs from a traffic schedule and coordinates them with a Kubernetes in-place Pod resize, which the JVM has no way to do on its own.
The shrink and grow sequences are not symmetric. Get the order wrong and you
OOMKilled the pod. This ordering is the core correctness guarantee of the
tool:
sequenceDiagram
participant Sched as Traffic Schedule
participant W as Warden Agent
participant JVM
participant K8s as Kubelet (cgroup)
Note over Sched,K8s: SHRINK — off-peak window begins
Sched->>W: low-traffic window predicted
W->>JVM: lower SoftMaxHeapSize / flush evictable cache
W->>JVM: force deep GC + uncommit pages
JVM-->>W: RSS dropped, pages returned to OS
W->>W: verify actual RSS < target limit ✅ gate
W->>K8s: in-place resize DOWN (request + limit)
Note right of W: cgroup lowered ONLY after RSS confirmed
Note over Sched,K8s: GROW — traffic returning
Sched->>W: peak window predicted
W->>K8s: in-place resize UP (request + limit) FIRST
K8s-->>W: headroom granted
W->>JVM: raise SoftMaxHeapSize
W->>JVM: (optional) pre-warm cache ahead of load
- Shrinking: shrink the JVM first, verify RSS actually fell, then lower the cgroup. Lowering the cgroup before the JVM has released memory = instant OOMKill.
- Growing: raise the cgroup first, then raise the JVM's soft max. Give the headroom before the JVM tries to use it.
- Cache re-warming happens ahead of the predicted traffic return — a flushed cache is itself a cold start, so Warden shrinks early and re-warms early rather than reactively.
Warden's real neighbor isn't HPA — it's the Vertical Pod Autoscaler, which now also supports in-place updates. The difference is that VPA is JVM-blind:
| VPA | mnemo-jvm-warden | |
|---|---|---|
| Resizes the Pod in place | ✅ | ✅ |
| Can cause the JVM to release memory | ❌ observes RSS, can't move it | ✅ drives soft-max, GC, uncommit |
| Coordinates GC/uncommit with the resize | ❌ | ✅ ordered handshake |
| Understands cache / working-set state | ❌ | ✅ flushes evictable, keeps hot set |
| Driven by predicted traffic curves | ❌ reactive to observed usage | ✅ proactive, schedule-aware |
VPA resizes the box and hopes the JVM cooperates. Warden makes the JVM cooperate first, then resizes.
Two charts, two different jobs. Both are published to a Helm repo hosted on GitHub Pages, so no local checkout is needed:
helm repo add warden https://baokhang83.github.io/mnemo-jvm-warden/
helm repo update(You can still install straight from a local checkout — helm install warden charts/warden —
if you're working on the charts themselves.)
helm install warden warden/wardenThis is the one-shot, cluster-wide install: it applies the WardenPolicy CRD (once — Helm's
crds/ directory is never touched again by helm upgrade, see charts/warden/crds/), a
ClusterRole/ClusterRoleBinding scoped to exactly what the controller reads and writes (watch
WardenPolicy, patch its status, read Deployment/StatefulSet, read/patch Pod
annotations), and the controller Deployment itself (hardcoded to replicas: 1 — there's no
leader election yet, so a second replica would double-reconcile every policy).
Values worth knowing (charts/warden/values.yaml):
| key | what it controls |
|---|---|
controller.image.repository / .tag |
which controller image to run |
controller.prometheusUrl |
where Prometheus lives, for guardrail metric evaluation (W-401) — leave empty if no policy uses a guardrail |
controller.resources |
the controller Pod's own request/limit (defaults to 256Mi request / 512Mi limit — the controller is a JVM with a heavy dependency graph, so its non-heap memory needs real headroom; 256Mi was too tight and OOMKilled it under load) |
There is no admission webhook. "Injecting" the sidecar means an operator's own app chart
includes a Helm named template that renders the same native-sidecar shape as
deploy/example-sidecar.yaml, parameterized instead of hand-copied. It's a Helm library
chart (type: library) on purpose: depending on it can never deploy a second controller
alongside your app, because a library chart is structurally forbidden from rendering any
resources of its own.
In your own app chart's Chart.yaml:
dependencies:
- name: warden-sidecar
version: "0.1.4" # match the published chart version (see `helm search repo warden`)
repository: "https://baokhang83.github.io/mnemo-jvm-warden/"In your own values.yaml, a warden: block shaped like charts/warden/values.yaml's
sidecar: section (targetContainerName and resources are required, with no default — see
that file's comments for why):
warden:
enabled: true
image:
repository: ghcr.io/baokhang83/mnemo-jvm-warden
tag: latest
targetContainerName: app # the sibling container Warden resizes
resources:
requests: { cpu: 25m, memory: 64Mi }
limits: { memory: 256Mi }And in your own Pod template:
spec:
shareProcessNamespace: true # lets the sidecar see the target JVM's PID (W-102)
volumes:
{{- include "warden.sidecar.volumes" . | nindent 4 }}
initContainers:
{{- if .Values.warden.enabled }}
{{- include "warden.sidecar" (dict "cfg" .Values.warden) | nindent 4 }}
{{- end }}
containers:
- name: app # must match warden.targetContainerName above
# ... your app container, with the JMX flags deploy/example-sidecar.yaml documentswarden.sidecar renders only the initContainer entry; shareProcessNamespace and the
host-cgroup volume are set by your own chart rather than injected, so your chart stays in full
control of its own Pod spec end to end.
Neither chart above makes Warden do anything on its own — it acts only once a WardenPolicy
exists for the target workload:
apiVersion: warden.mnemo.io/v1alpha1
kind: WardenPolicy
metadata:
name: my-app-policy
spec:
targetRef:
apiVersion: apps/v1
kind: Deployment
name: my-app
timezone: Europe/Paris
profiles:
off-peak: { request: 512Mi, limit: 768Mi }
peak: { request: 2Gi, limit: 3Gi }
schedule:
- { cron: "0 22 * * *", profile: off-peak }
- { cron: "0 7 * * *", profile: peak }Apply it with kubectl apply -f, same as any other custom resource. For every field
(leadTime, blackout, guardrail), the sidecar/controller environment variables, and the
full set of Helm values, see docs/configuration.md — it also has
an operator runbook for the failure modes you'll actually hit (read-only mode on an
unsupported collector, a shrink aborted by the RSS verification gate, cgroup mount issues, and
more).
🧪 Beta. The controller, agent, CRD, and both Helm charts are built, released, and installable (published container images on GHCR, charts on the Helm repo above, libraries on Maven Central), and the full schedule → shrink/grow → verify flow works end to end against a real cluster. It has not yet been battle-tested under production load, and APIs may still change — issues and feedback welcome.
- Kubernetes 1.35+ (in-place Pod resize is GA; on earlier versions it needs the
InPlacePodVerticalScalingfeature gate) - A JVM with runtime-tunable heap commit: ZGC, Shenandoah, or G1 with periodic GC enabled
- Cluster Autoscaler (or equivalent node consolidation) to realize cost savings from reduced requests
Warden runs with real privileges inside your cluster (RBAC, JMX, a host cgroup mount). See SECURITY.md for how to report a vulnerability, docs/threat-model.md for the full trust-boundary analysis, and docs/hardening.md for an operator checklist before going to production.