PoC: share PMM's built-in PostgreSQL with a side container

dev/pg-share-proof/README.md

@@ -0,0 +1,71 @@
+# Proof: sharing PMM's built-in PostgreSQL with a side container
+
+Two Docker Compose files, identical topology — **two separate containers on a shared
+user-defined bridge network** (separate network namespaces; no `network_mode: "container:"`).
+The only difference is whether PMM's PG is opened up. The same probe tool runs in both.
+
+The variable being isolated is `listen_addresses` (in `postgresql.conf`), which is what the
+"listen on all interfaces" question is actually about — NOT `pg_hba.conf`.
+
+## Fail path — `pg-share-fail.compose.yml`
+PMM's PG left untouched → binds `127.0.0.1` only. The client hits pmm-server's `eth0`, where
+nothing listens → connection refused.
+
+```
+docker compose -f pg-share-fail.compose.yml up --abort-on-container-exit --exit-code-from client
+docker compose -f pg-share-fail.compose.yml down -v   # cleanup
+```
+Expected client output: `pg_isready` → `no response` (rc=2); `psql` → `Connection refused`.
+
+## Success path — `pg-share-success.compose.yml` (+ `run-success.sh`)
+Same setup, plus a runtime edit inside pmm-server: append `listen_addresses = '*'` to
+`/srv/postgres14/postgresql.conf` and a `host ... scram-sha-256` rule to
+`/srv/postgres14/pg_hba.conf`, then `supervisorctl restart postgresql`. The client (separate
+container) then connects over the bridge.
+
+```
+./run-success.sh
+docker compose -f pg-share-success.compose.yml --profile probe down -v   # cleanup
+```
+Expected client output: `pg_isready` → `accepting connections` (rc=0); `psql` returns
+`pmm-managed` and the bridge server IP, then lists PMM tables.
+
+## Why `EXPOSE`/`ports:` can't substitute for the edit
+Neither file publishes 5432 to the host, and neither needs to. The fail case proves that a
+localhost-bound service is unreachable from another container regardless of any Compose port
+directive — `EXPOSE`/`expose:` is metadata only, and even `ports:` DNATs to the container's
+bridge IP (its `eth0`), which a `127.0.0.1`-bound PG still refuses. Reachability is decided by
+where the *process* binds, i.e. `listen_addresses`.
+
+## Why `network_mode: "service:pmm"` (shared namespace) is the wrong fix — and a security issue
+A tempting shortcut is to put the side container in PMM's network namespace
+(`network_mode: "service:pmm"`, i.e. the Compose form of `--network container:pmm-server`).
+It "works" on `127.0.0.1:5432` with **zero** PMM changes — but only because it doesn't grant
+access to Postgres, it moves the client *inside* PMM's trust boundary. That's a security
+problem, not a solution:
+
+- **It exposes everything PMM keeps private, not just Postgres.** PMM binds its internal
+  services to `127.0.0.1` precisely so nothing else can reach them (ClickHouse, VictoriaMetrics,
+  internal HTTP endpoints, …). Sharing the namespace gives the sidecar loopback-level access to
+  *all* of them — including services that trust localhost implicitly and have no auth. The
+  localhost bind **is** the security boundary; a shared namespace collapses it wholesale, with no
+  way to scope access to only the DB.
+- **No isolation or auditability.** The sidecar has no network identity of its own, so this
+  access is invisible to any network policy, firewall, or segmentation — it can't be limited to a
+  subnet/user or observed as a real endpoint connection.
+- **It couples the two containers** (shared port space → collisions on 8080/8443/5432/9090/…;
+  the sidecar's networking is tied to PMM's lifecycle) and **doesn't generalize** beyond a single
+  Docker host (a separate pod/host/network needs a real endpoint anyway).
+
+The supported, least-privilege alternative is the "success path" above: keep the containers
+independently networked, bind PG to a reachable interface (`listen_addresses`), and authorize
+**only** the intended client subnet/user (`pg_hba`). Everything else stays private.
+
+## Notes / caveats
+- Separate named volumes + project names per file, so the fail case is genuinely unconfigured.
+- `0.0.0.0/0` in the pg_hba rule is demo-only; scope to the bridge subnet
+  (`docker network inspect pmm-pg-success_pmmnet -f '{{range .IPAM.Config}}{{.Subnet}}{{end}}'`).
+- The demo reuses the `pmm-managed` role/DB for brevity. For real use, create a dedicated
+  least-privilege role + database instead of handing a side app PMM's owner role.
+- Uses `percona/pmm-server:3.8.1`, pinned to `platform: linux/amd64` (no arm64 build yet, so
+  it runs emulated on Apple Silicon — expect a slower boot, hence the generous healthcheck).

dev/pg-share-proof/pg-share-fail.compose.yml

@@ -0,0 +1,50 @@
+# FAIL PATH — PMM's built-in PostgreSQL is left UNCHANGED.
+# It binds 127.0.0.1 only (initdb default; PMM sets no listen_addresses).
+# A separate client container on the shared bridge hits pmm-server's eth0,
+# where nothing is listening -> "Connection refused".
+#
+# Run (turnkey, no exec needed):
+#   docker compose -f pg-share-fail.compose.yml up --abort-on-container-exit --exit-code-from client
+# Cleanup:
+#   docker compose -f pg-share-fail.compose.yml down -v
+name: pmm-pg-fail
+
+networks:
+  pmmnet:
+    driver: bridge
+
+volumes:
+  pmm-fail-srv:
+
+services:
+  pmm-server:
+    image: percona/pmm-server:3.8.1
+    # no arm64 build yet -> force amd64 (emulated on Apple Silicon)
+    platform: linux/amd64
+    hostname: pmm-server
+    networks: [pmmnet]
+    volumes:
+      - pmm-fail-srv:/srv
+    healthcheck:
+      # local socket is trust auth -> no password; gates until PG is initialized and up
+      test: ["CMD-SHELL", "/usr/bin/psql -U postgres -h /run/postgresql -d postgres -tAc 'select 1' >/dev/null 2>&1"]
+      interval: 5s
+      timeout: 10s
+      retries: 60
+      start_period: 180s
+
+  client:
+    image: postgres:16-alpine
+    networks: [pmmnet]
+    depends_on:
+      pmm-server:
+        condition: service_healthy
+    entrypoint: ["/bin/sh", "-c"]
+    command:
+      - |
+        echo "=== FAIL CASE: PMM PG unchanged (binds 127.0.0.1 only) ==="
+        echo "--- TCP reachability probe (pg_isready to pmm-server:5432) ---"
+        pg_isready -h pmm-server -p 5432; echo "pg_isready rc=$$?"
+        echo "--- Full connect attempt (psql, same tool as success case) ---"
+        psql "postgres://pmm-managed:pmm-managed@pmm-server:5432/pmm-managed?sslmode=disable" -c '\dt' || echo "psql failed rc=$$?"
+        echo "=== EXPECTED: 'no response' / 'Connection refused' — PG is not listening on the bridge interface ==="

dev/pg-share-proof/pg-share-success.compose.yml

@@ -0,0 +1,60 @@
+# SUCCESS PATH — same topology (separate containers, shared bridge, NO shared
+# network namespace). The difference vs the fail file is a runtime edit applied
+# INSIDE pmm-server: listen_addresses='*' (in postgresql.conf) + a scram host
+# rule (in pg_hba.conf), then a PG restart. The client (a separate container)
+# then connects over the bridge.
+#
+# The edit must run inside pmm-server after PG init, so use the helper:
+#   ./run-success.sh
+# or do it manually:
+#   docker compose -f pg-share-success.compose.yml up -d pmm-server
+#   # wait until healthy, then:
+#   docker compose -f pg-share-success.compose.yml exec pmm-server bash -c \
+#     "echo \"listen_addresses = '*'\" >> /srv/postgres14/postgresql.conf; \
+#      echo 'host pmm-managed pmm-managed 0.0.0.0/0 scram-sha-256' >> /srv/postgres14/pg_hba.conf; \
+#      supervisorctl restart postgresql"
+#   docker compose -f pg-share-success.compose.yml --profile probe run --rm client
+# Cleanup:
+#   docker compose -f pg-share-success.compose.yml --profile probe down -v
+name: pmm-pg-success
+
+networks:
+  pmmnet:
+    driver: bridge
+
+volumes:
+  pmm-success-srv:
+
+services:
+  pmm-server:
+    image: percona/pmm-server:3.8.1
+    # no arm64 build yet
+    platform: linux/amd64
+    hostname: pmm-server
+    networks: [pmmnet]
+    volumes:
+      - pmm-success-srv:/srv
+    healthcheck:
+      test: ["CMD-SHELL", "/usr/bin/psql -U postgres -h /run/postgresql -d postgres -tAc 'select 1' >/dev/null 2>&1"]
+      interval: 5s
+      timeout: 10s
+      retries: 60
+      start_period: 180s
+
+  # gated behind a profile so `up` starts only pmm-server; run explicitly after the edit
+  client:
+    image: postgres:16-alpine
+    networks: [pmmnet]
+    profiles: ["probe"]
+    entrypoint: ["/bin/sh", "-c"]
+    command:
+      - |
+        echo "=== SUCCESS CASE: PMM PG now listens on '*' ==="
+        echo "--- TCP reachability probe (pg_isready to pmm-server:5432) ---"
+        pg_isready -h pmm-server -p 5432; echo "pg_isready rc=$$?"
+        echo "--- Full authenticated query (psql over the bridge) ---"
+        psql "postgres://pmm-managed:pmm-managed@pmm-server:5432/pmm-managed?sslmode=disable" \
+          -tAc "select current_user, inet_server_addr() as connected_to_server_ip;"
+        echo "--- A few PMM tables, proving it's the real pmm-managed DB ---"
+        psql "postgres://pmm-managed:pmm-managed@pmm-server:5432/pmm-managed?sslmode=disable" -c '\dt' | head -n 15
+        echo "=== EXPECTED: accepting connections; query returns pmm-managed + the bridge IP ==="

dev/pg-share-proof/run-success.sh

@@ -0,0 +1,43 @@
+#!/usr/bin/env bash
+# Turnkey driver for the SUCCESS path:
+# 1. boot pmm-server
+# 2. wait for its built-in PostgreSQL
+# 3. apply the runtime edit (listen_addresses='*' + pg_hba host rule)
+# 4. restart PG
+# 5. run the client probe from a SEPARATE container on the bridge.
+set -euo pipefail
+
+DIR="$(cd "$(dirname "$0")" && pwd)"
+F="$DIR/pg-share-success.compose.yml"
+
+echo "[1/4] Starting pmm-server (this pulls the image on first run)..."
+docker compose -f "$F" up -d pmm-server
+
+echo "[2/4] Waiting for built-in PostgreSQL to be ready..."
+CID="$(docker compose -f "$F" ps -q pmm-server)"
+until [ "$(docker inspect -f '{{.State.Health.Status}}' "$CID")" = "healthy" ]; do
+  sleep 3
+  echo "  ...still starting"
+done
+echo "  pmm-server is healthy."
+
+echo "[3/4] Applying runtime edit + restarting PG (inside pmm-server)..."
+docker compose -f "$F" exec -T pmm-server bash -s <<'REMOTE'
+set -e
+CONF=/srv/postgres14/postgresql.conf
+HBA=/srv/postgres14/pg_hba.conf
+grep -Eq "^listen_addresses = '\*'" "$CONF" || printf "\nlisten_addresses = '*'\n" >> "$CONF"
+# scram (NOT trust) so the on-boot pg_hba migration leaves it intact.
+# 0.0.0.0/0 is for the demo only — scope to the bridge subnet in real use.
+grep -q "0.0.0.0/0" "$HBA" || printf "host    pmm-managed    pmm-managed    0.0.0.0/0    scram-sha-256\n" >> "$HBA"
+supervisorctl restart postgresql
+sleep 4
+echo -n "  effective listen_addresses on server: "
+/usr/bin/psql -U postgres -h /run/postgresql -d postgres -tAc "show listen_addresses;"
+REMOTE
+
+echo "[4/4] Running client probe from a separate container on the shared bridge..."
+docker compose -f "$F" --profile probe run --rm client
+
+echo
+echo "Done. Cleanup with: docker compose -f \"$F\" --profile probe down -v"