diff --git a/dev/pg-share-proof/README.md b/dev/pg-share-proof/README.md new file mode 100644 index 00000000000..23aae9d8e67 --- /dev/null +++ b/dev/pg-share-proof/README.md @@ -0,0 +1,71 @@ +# Proof: sharing PMM's built-in PostgreSQL with a side container + +Two Docker Compose files, identical topology — **two separate containers on a shared +user-defined bridge network** (separate network namespaces; no `network_mode: "container:"`). +The only difference is whether PMM's PG is opened up. The same probe tool runs in both. + +The variable being isolated is `listen_addresses` (in `postgresql.conf`), which is what the +"listen on all interfaces" question is actually about — NOT `pg_hba.conf`. + +## Fail path — `pg-share-fail.compose.yml` +PMM's PG left untouched → binds `127.0.0.1` only. The client hits pmm-server's `eth0`, where +nothing listens → connection refused. + +``` +docker compose -f pg-share-fail.compose.yml up --abort-on-container-exit --exit-code-from client +docker compose -f pg-share-fail.compose.yml down -v # cleanup +``` +Expected client output: `pg_isready` → `no response` (rc=2); `psql` → `Connection refused`. + +## Success path — `pg-share-success.compose.yml` (+ `run-success.sh`) +Same setup, plus a runtime edit inside pmm-server: append `listen_addresses = '*'` to +`/srv/postgres14/postgresql.conf` and a `host ... scram-sha-256` rule to +`/srv/postgres14/pg_hba.conf`, then `supervisorctl restart postgresql`. The client (separate +container) then connects over the bridge. + +``` +./run-success.sh +docker compose -f pg-share-success.compose.yml --profile probe down -v # cleanup +``` +Expected client output: `pg_isready` → `accepting connections` (rc=0); `psql` returns +`pmm-managed` and the bridge server IP, then lists PMM tables. + +## Why `EXPOSE`/`ports:` can't substitute for the edit +Neither file publishes 5432 to the host, and neither needs to. The fail case proves that a +localhost-bound service is unreachable from another container regardless of any Compose port +directive — `EXPOSE`/`expose:` is metadata only, and even `ports:` DNATs to the container's +bridge IP (its `eth0`), which a `127.0.0.1`-bound PG still refuses. Reachability is decided by +where the *process* binds, i.e. `listen_addresses`. + +## Why `network_mode: "service:pmm"` (shared namespace) is the wrong fix — and a security issue +A tempting shortcut is to put the side container in PMM's network namespace +(`network_mode: "service:pmm"`, i.e. the Compose form of `--network container:pmm-server`). +It "works" on `127.0.0.1:5432` with **zero** PMM changes — but only because it doesn't grant +access to Postgres, it moves the client *inside* PMM's trust boundary. That's a security +problem, not a solution: + +- **It exposes everything PMM keeps private, not just Postgres.** PMM binds its internal + services to `127.0.0.1` precisely so nothing else can reach them (ClickHouse, VictoriaMetrics, + internal HTTP endpoints, …). Sharing the namespace gives the sidecar loopback-level access to + *all* of them — including services that trust localhost implicitly and have no auth. The + localhost bind **is** the security boundary; a shared namespace collapses it wholesale, with no + way to scope access to only the DB. +- **No isolation or auditability.** The sidecar has no network identity of its own, so this + access is invisible to any network policy, firewall, or segmentation — it can't be limited to a + subnet/user or observed as a real endpoint connection. +- **It couples the two containers** (shared port space → collisions on 8080/8443/5432/9090/…; + the sidecar's networking is tied to PMM's lifecycle) and **doesn't generalize** beyond a single + Docker host (a separate pod/host/network needs a real endpoint anyway). + +The supported, least-privilege alternative is the "success path" above: keep the containers +independently networked, bind PG to a reachable interface (`listen_addresses`), and authorize +**only** the intended client subnet/user (`pg_hba`). Everything else stays private. + +## Notes / caveats +- Separate named volumes + project names per file, so the fail case is genuinely unconfigured. +- `0.0.0.0/0` in the pg_hba rule is demo-only; scope to the bridge subnet + (`docker network inspect pmm-pg-success_pmmnet -f '{{range .IPAM.Config}}{{.Subnet}}{{end}}'`). +- The demo reuses the `pmm-managed` role/DB for brevity. For real use, create a dedicated + least-privilege role + database instead of handing a side app PMM's owner role. +- Uses `percona/pmm-server:3.8.1`, pinned to `platform: linux/amd64` (no arm64 build yet, so + it runs emulated on Apple Silicon — expect a slower boot, hence the generous healthcheck). diff --git a/dev/pg-share-proof/pg-share-fail.compose.yml b/dev/pg-share-proof/pg-share-fail.compose.yml new file mode 100644 index 00000000000..89d21b59bf8 --- /dev/null +++ b/dev/pg-share-proof/pg-share-fail.compose.yml @@ -0,0 +1,50 @@ +# FAIL PATH — PMM's built-in PostgreSQL is left UNCHANGED. +# It binds 127.0.0.1 only (initdb default; PMM sets no listen_addresses). +# A separate client container on the shared bridge hits pmm-server's eth0, +# where nothing is listening -> "Connection refused". +# +# Run (turnkey, no exec needed): +# docker compose -f pg-share-fail.compose.yml up --abort-on-container-exit --exit-code-from client +# Cleanup: +# docker compose -f pg-share-fail.compose.yml down -v +name: pmm-pg-fail + +networks: + pmmnet: + driver: bridge + +volumes: + pmm-fail-srv: + +services: + pmm-server: + image: percona/pmm-server:3.8.1 + # no arm64 build yet -> force amd64 (emulated on Apple Silicon) + platform: linux/amd64 + hostname: pmm-server + networks: [pmmnet] + volumes: + - pmm-fail-srv:/srv + healthcheck: + # local socket is trust auth -> no password; gates until PG is initialized and up + test: ["CMD-SHELL", "/usr/bin/psql -U postgres -h /run/postgresql -d postgres -tAc 'select 1' >/dev/null 2>&1"] + interval: 5s + timeout: 10s + retries: 60 + start_period: 180s + + client: + image: postgres:16-alpine + networks: [pmmnet] + depends_on: + pmm-server: + condition: service_healthy + entrypoint: ["/bin/sh", "-c"] + command: + - | + echo "=== FAIL CASE: PMM PG unchanged (binds 127.0.0.1 only) ===" + echo "--- TCP reachability probe (pg_isready to pmm-server:5432) ---" + pg_isready -h pmm-server -p 5432; echo "pg_isready rc=$$?" + echo "--- Full connect attempt (psql, same tool as success case) ---" + psql "postgres://pmm-managed:pmm-managed@pmm-server:5432/pmm-managed?sslmode=disable" -c '\dt' || echo "psql failed rc=$$?" + echo "=== EXPECTED: 'no response' / 'Connection refused' — PG is not listening on the bridge interface ===" diff --git a/dev/pg-share-proof/pg-share-success.compose.yml b/dev/pg-share-proof/pg-share-success.compose.yml new file mode 100644 index 00000000000..2acd1c24372 --- /dev/null +++ b/dev/pg-share-proof/pg-share-success.compose.yml @@ -0,0 +1,60 @@ +# SUCCESS PATH — same topology (separate containers, shared bridge, NO shared +# network namespace). The difference vs the fail file is a runtime edit applied +# INSIDE pmm-server: listen_addresses='*' (in postgresql.conf) + a scram host +# rule (in pg_hba.conf), then a PG restart. The client (a separate container) +# then connects over the bridge. +# +# The edit must run inside pmm-server after PG init, so use the helper: +# ./run-success.sh +# or do it manually: +# docker compose -f pg-share-success.compose.yml up -d pmm-server +# # wait until healthy, then: +# docker compose -f pg-share-success.compose.yml exec pmm-server bash -c \ +# "echo \"listen_addresses = '*'\" >> /srv/postgres14/postgresql.conf; \ +# echo 'host pmm-managed pmm-managed 0.0.0.0/0 scram-sha-256' >> /srv/postgres14/pg_hba.conf; \ +# supervisorctl restart postgresql" +# docker compose -f pg-share-success.compose.yml --profile probe run --rm client +# Cleanup: +# docker compose -f pg-share-success.compose.yml --profile probe down -v +name: pmm-pg-success + +networks: + pmmnet: + driver: bridge + +volumes: + pmm-success-srv: + +services: + pmm-server: + image: percona/pmm-server:3.8.1 + # no arm64 build yet + platform: linux/amd64 + hostname: pmm-server + networks: [pmmnet] + volumes: + - pmm-success-srv:/srv + healthcheck: + test: ["CMD-SHELL", "/usr/bin/psql -U postgres -h /run/postgresql -d postgres -tAc 'select 1' >/dev/null 2>&1"] + interval: 5s + timeout: 10s + retries: 60 + start_period: 180s + + # gated behind a profile so `up` starts only pmm-server; run explicitly after the edit + client: + image: postgres:16-alpine + networks: [pmmnet] + profiles: ["probe"] + entrypoint: ["/bin/sh", "-c"] + command: + - | + echo "=== SUCCESS CASE: PMM PG now listens on '*' ===" + echo "--- TCP reachability probe (pg_isready to pmm-server:5432) ---" + pg_isready -h pmm-server -p 5432; echo "pg_isready rc=$$?" + echo "--- Full authenticated query (psql over the bridge) ---" + psql "postgres://pmm-managed:pmm-managed@pmm-server:5432/pmm-managed?sslmode=disable" \ + -tAc "select current_user, inet_server_addr() as connected_to_server_ip;" + echo "--- A few PMM tables, proving it's the real pmm-managed DB ---" + psql "postgres://pmm-managed:pmm-managed@pmm-server:5432/pmm-managed?sslmode=disable" -c '\dt' | head -n 15 + echo "=== EXPECTED: accepting connections; query returns pmm-managed + the bridge IP ===" diff --git a/dev/pg-share-proof/run-success.sh b/dev/pg-share-proof/run-success.sh new file mode 100755 index 00000000000..9fc9f5342c4 --- /dev/null +++ b/dev/pg-share-proof/run-success.sh @@ -0,0 +1,43 @@ +#!/usr/bin/env bash +# Turnkey driver for the SUCCESS path: +# 1. boot pmm-server +# 2. wait for its built-in PostgreSQL +# 3. apply the runtime edit (listen_addresses='*' + pg_hba host rule) +# 4. restart PG +# 5. run the client probe from a SEPARATE container on the bridge. +set -euo pipefail + +DIR="$(cd "$(dirname "$0")" && pwd)" +F="$DIR/pg-share-success.compose.yml" + +echo "[1/4] Starting pmm-server (this pulls the image on first run)..." +docker compose -f "$F" up -d pmm-server + +echo "[2/4] Waiting for built-in PostgreSQL to be ready..." +CID="$(docker compose -f "$F" ps -q pmm-server)" +until [ "$(docker inspect -f '{{.State.Health.Status}}' "$CID")" = "healthy" ]; do + sleep 3 + echo " ...still starting" +done +echo " pmm-server is healthy." + +echo "[3/4] Applying runtime edit + restarting PG (inside pmm-server)..." +docker compose -f "$F" exec -T pmm-server bash -s <<'REMOTE' +set -e +CONF=/srv/postgres14/postgresql.conf +HBA=/srv/postgres14/pg_hba.conf +grep -Eq "^listen_addresses = '\*'" "$CONF" || printf "\nlisten_addresses = '*'\n" >> "$CONF" +# scram (NOT trust) so the on-boot pg_hba migration leaves it intact. +# 0.0.0.0/0 is for the demo only — scope to the bridge subnet in real use. +grep -q "0.0.0.0/0" "$HBA" || printf "host pmm-managed pmm-managed 0.0.0.0/0 scram-sha-256\n" >> "$HBA" +supervisorctl restart postgresql +sleep 4 +echo -n " effective listen_addresses on server: " +/usr/bin/psql -U postgres -h /run/postgresql -d postgres -tAc "show listen_addresses;" +REMOTE + +echo "[4/4] Running client probe from a separate container on the shared bridge..." +docker compose -f "$F" --profile probe run --rm client + +echo +echo "Done. Cleanup with: docker compose -f \"$F\" --profile probe down -v"