llms.txt

# Prometheus Proxy

> Prometheus Proxy enables Prometheus to scrape metrics from endpoints behind firewalls while preserving the native pull-based model. It uses a persistent gRPC connection initiated from inside the firewall, so only outbound access is needed from the agent side.

The system has two components: a **proxy** that runs alongside Prometheus (outside the firewall) and an **agent** that runs alongside monitored services (inside the firewall). Prometheus scrapes the proxy as if it were the real endpoint; the proxy forwards the request to the agent over gRPC; the agent scrapes the actual metrics endpoint and returns the response.

## Links

- [Documentation](https://pambrose.github.io/prometheus-proxy/)
- [GitHub Repository](https://github.com/pambrose/prometheus-proxy)
- [README](https://github.com/pambrose/prometheus-proxy/blob/master/README.md)
- [Changelog](https://github.com/pambrose/prometheus-proxy/blob/master/CHANGELOG.md)
- [Release Notes](https://github.com/pambrose/prometheus-proxy/blob/master/RELEASE_NOTES.md)
- [KDoc API Reference](https://pambrose.github.io/prometheus-proxy/kdocs/)
- [CLI Reference](https://pambrose.github.io/prometheus-proxy/cli-reference/)
- [Maven Central](https://central.sonatype.com/artifact/com.pambrose/prometheus-proxy)
- [Docker Images: pambrose/prometheus-proxy](https://hub.docker.com/r/pambrose/prometheus-proxy)
- [Docker Images: pambrose/prometheus-agent](https://hub.docker.com/r/pambrose/prometheus-agent)

## Architecture

```
Prometheus --> Proxy HTTP (:8080) --> AgentContext lookup --> ScrapeRequest via gRPC stream
    --> Agent scrapes actual endpoint --> ScrapeResponse via gRPC stream --> Proxy --> Prometheus
```

### Proxy (runs outside firewall)

- HTTP server on port 8080: serves proxied metrics to Prometheus
- gRPC server on port 50051: accepts persistent agent connections
- `ProxyGrpcService` accepts agent connections
- `ProxyHttpService` / `ProxyHttpRoutes` handle HTTP scrape requests from Prometheus
- `ProxyPathManager` maps URL paths to agent contexts
- `AgentContextManager` tracks connected agents
- `ScrapeRequestManager` manages scrape request lifecycle with timeouts
- `ProxyServiceImpl` implements the gRPC `ProxyService`

### Agent (runs inside firewall)

- Connects outbound to proxy via gRPC (no inbound ports needed)
- `AgentGrpcService` manages the gRPC connection and streams
- `AgentHttpService` scrapes actual metrics endpoints using Ktor HTTP client
- `AgentPathManager` manages path registrations
- `HttpClientCache` caches HTTP clients keyed by auth credentials (TTL/idle eviction)

### gRPC Service (proxy_service.proto)

Key RPCs:
- `registerAgent` / `registerPath` / `unregisterPath` - agent registration lifecycle
- `readRequestsFromProxy` - server-streaming: proxy sends scrape requests to agent
- `writeResponsesToProxy` - client-streaming: agent sends scrape responses back
- `writeChunkedResponsesToProxy` - client-streaming: chunked responses for large payloads (>32KB default)
- `sendHeartBeat` - keepalive during inactivity (default 5s)

### Key Mechanisms

- **Chunking**: Large metric payloads are split into `ChunkedScrapeResponse` messages to stay within gRPC limits (configurable via `chunkContentSizeKbs`, default 32KB)
- **Stale agent cleanup**: `AgentContextCleanupService` evicts inactive agents after `maxAgentInactivitySecs` (default 60s)
- **Consolidated mode**: Multiple agents can register the same path for redundancy
- **Auth forwarding**: Proxy forwards `Authorization` headers (basic auth / bearer token) from Prometheus scrape configs to agents over gRPC
- **Embedded agent**: Agents can run inside other JVM apps via `Agent.startAsyncAgent()`, returning an `EmbeddedAgentInfo` lifecycle handle
- **Proxy failover** (4.0.0): The agent takes an ordered `agent.proxy.endpoints` list (or comma-separated `--proxy` / `PROXY_HOSTNAME`); a failed connect advances to the next endpoint, a dropped connection retries from the head, so a recovered primary is picked up automatically. Scrape an HA pair with `static_config`, not `http_sd_config` (a standby returns an empty discovery list, which Prometheus treats as target deletion)
- **Dynamic target discovery** (4.0.0): Opt-in `agent.discovery` reconcile loop keeps registered paths in sync with a watched HOCON/JSON file (`paths` list of `{ name, path, url, labels }`), so targets change at runtime with no agent restart. Static `pathConfigs` are never touched; a read failure keeps the last-known-good set, while a valid-but-empty file removes all discovered paths
- **Per-agent identities** (4.0.0): `proxy.auth` lists named identities (token + allowed path globs) enforced on every `registerPath`, closing the shared-token hole where any agent could register (and take over) any path. Legacy `proxy.agentToken` is honored as an allow-all identity for migration
- **Metric filtering** (4.0.0): Optional per-path `agent.filters` (fully-anchored `metricNameAllow` / `metricNameDeny` regexes) drop whole metric families at the agent before gzip/chunking, so unwanted series never cross the WAN. Families are kept or dropped atomically (a histogram's `_bucket`/`_sum`/`_count` stay together); non-text or non-UTF-8 payloads pass through unfiltered (fails open)
- **Operational dashboard** (4.0.0): Read-only live web UI on its own port (`--dashboard`, default port 8094, off by default, unauthenticated) showing connected agents, registered paths — including paths whose agent has gone — and recent scrape results. Two layouts: `/dashboard` (per-agent) and `/dashboard/paths` (one row per path). Both identify agents by name, not internal id, so the two layouts can be read against each other (4.0.1). The page meets WCAG AA (measured contrast floor 4.59:1) and announces connection loss and recovery through a live region (4.0.1); `DESIGN.md` records the visual system and the contrast floor

### Public API Surface

These are the only types intended for external consumption (everything else is `internal`):

- `io.prometheus.Agent` and `io.prometheus.Proxy` (entry points)
- `io.prometheus.agent.AgentOptions`, `io.prometheus.proxy.ProxyOptions`, `io.prometheus.common.BaseOptions` (config/CLI parsing)
- `io.prometheus.agent.EmbeddedAgentInfo` (returned by `Agent.startAsyncAgent`)
- `io.prometheus.common.EnvVars` (typed env-var enum)
- `io.prometheus.common.ConfigLoadException` (thrown by `startAsyncAgent` on a config-load failure when `exitOnMissingConfig` is false, so embedded hosts catch it instead of the JVM exiting)

## Quick Start

Requirements: Java 17+

### CLI

```bash
# Start proxy
java -jar prometheus-proxy.jar

# Start agent (pointing to proxy)
java -jar prometheus-agent.jar --proxy mymachine.local --config myapps.conf
```

### Docker

```bash
# Start proxy
docker run --rm -p 8080:8080 -p 50051:50051 pambrose/prometheus-proxy:4.0.1

# Start agent
docker run --rm \
  --env AGENT_CONFIG='https://raw.githubusercontent.com/pambrose/prometheus-proxy/master/examples/simple.conf' \
  pambrose/prometheus-agent:4.0.1
```

## Configuration

Uses Typesafe Config (HOCON). Precedence: CLI args > environment variables > config file values.

### Agent Path Configuration (HOCON)

```hocon
agent {
  pathConfigs: [
    {
      name: "App1 metrics"
      path: app1_metrics
      url: "http://app1.local:9100/metrics"
      labels: "{\"key1\": \"value1\"}"
    }
  ]
}
```

### Common Proxy Options

| Option | Env Var | Default | Description |
|--------|---------|---------|-------------|
| `--port, -p` | `PROXY_PORT` | 8080 | Proxy listen port |
| `--agent_port, -a` | `AGENT_PORT` | 50051 | gRPC listen port for agents |
| `--agent_token` | `AGENT_TOKEN` | | Pre-shared agent auth token; agents must present the same value (empty disables) |
| `--config, -c` | `PROXY_CONFIG` | | Config file or URL |
| `--admin, -r` | `ADMIN_ENABLED` | false | Enable admin endpoints |
| `--metrics, -e` | `METRICS_ENABLED` | false | Enable internal metrics |
| `--sd_enabled` | `SD_ENABLED` | false | Service discovery endpoint |
| `--dashboard` | `DASHBOARD_ENABLED` | false | Read-only operational dashboard (own port, default 8094) |
| `--cert, -t` | `CERT_CHAIN_FILE_PATH` | | TLS certificate chain |
| `--key, -k` | `PRIVATE_KEY_FILE_PATH` | | TLS private key |
| `--trust, -s` | `TRUST_CERT_COLLECTION_FILE_PATH` | | TLS trust certificates |

### Common Agent Options

| Option | Env Var | Default | Description |
|--------|---------|---------|-------------|
| `--proxy, -p` | `PROXY_HOSTNAME` | localhost | Proxy hostname; accepts a comma-separated failover list (`proxy-a:50051,proxy-b:50051`) |
| `--config, -c` | `AGENT_CONFIG` | | Config file or URL (required) |
| `--name, -n` | `AGENT_NAME` | | Agent name |
| `--consolidated, -o` | `CONSOLIDATED` | false | Multiple agents per path |
| `--agent_token` | `AGENT_TOKEN` | | Pre-shared token presented to the proxy; must match the proxy's value (empty disables) |
| `--timeout` | `SCRAPE_TIMEOUT_SECS` | 15 | Scrape timeout (seconds) |
| `--chunk` | `CHUNK_CONTENT_SIZE_KBS` | 32 | Chunking threshold (KB) |
| `--trust_all_x509` | `TRUST_ALL_X509_CERTIFICATES` | false | Disable SSL verification |
| `--https_truststore` | `HTTPS_TRUST_STORE_PATH` | | Trust store (JKS/PKCS12) for HTTPS scrape targets |
| `--https_truststore_password` | `HTTPS_TRUST_STORE_PASSWORD` | | Password for `--https_truststore` |
| `--max_concurrent_clients` | `MAX_CONCURRENT_CLIENTS` | 1 | Concurrent HTTP clients |

## TLS

TLS without mutual auth requires:
- Proxy: `certChainFilePath`, `privateKeyFilePath`
- Agent: `trustCertCollectionFilePath`

TLS with mutual auth additionally requires:
- Proxy: `trustCertCollectionFilePath`
- Agent: `certChainFilePath`, `privateKeyFilePath`

Scraping HTTPS targets (agent → endpoint, separate from the agent ↔ proxy gRPC TLS above):
- `agent.http.trustStorePath` (+ `trustStorePassword`) — trust a custom/private CA, validation stays on
- `agent.http.enableTrustAllX509Certificates = true` — disable verification entirely (dev only; takes precedence)

## Building from Source

```bash
git clone https://github.com/pambrose/prometheus-proxy.git
cd prometheus-proxy
./gradlew build                              # Build with tests
./gradlew agentJar proxyJar                  # Named fat JARs in build/libs/
./gradlew build -PoverrideVersion=X.Y.Z      # Build with a custom version
```

`group` and `version` live in `gradle.properties` (single source of truth) and are read by `build.gradle.kts` via `providers.gradleProperty(...)`. `BuildConfig.APP_RELEASE_DATE` and `BuildConfig.BUILD_TIME` are populated each build via `ValueSource`, so they reflect the actual build time and are not overridable. `agentJar` and `proxyJar` are dedicated `ShadowJar` tasks; the default `shadowJar` task is disabled to avoid producing a third redundant fat jar. Repository declarations are centralized in `settings.gradle.kts` (`FAIL_ON_PROJECT_REPOS`).

The fat JARs include `src/shadow/resources/META-INF/services/io.grpc.NameResolverProvider` and `io.grpc.LoadBalancerProvider` so DNS / PickFirst providers stay registered — shadow 9.4.2 silently drops one of the two same-named files when both `grpc-core` and `grpc-netty-shaded` contribute, and without DNS the gRPC client defaults to `unix:///host:port` on any non-IP hostname.

## Tech Stack

- **Language**: Kotlin (JVM 17+)
- **Build**: Gradle with Kotlin DSL
- **gRPC**: protobuf + grpc-java + grpc-kotlin
- **HTTP Server**: Ktor (Netty engine)
- **HTTP Client**: Ktor CIO client
- **Configuration**: Typesafe Config (HOCON)
- **Metrics**: Prometheus Java SimpleClient
- **Testing**: Kotest + JUnit 5 + MockK; Testcontainers smoke test for Docker image regressions
- **Code Quality**: kotlinter + detekt

## Maven Central

Published as `com.pambrose:prometheus-proxy`:

```kotlin
// build.gradle.kts
repositories {
  mavenCentral()
}

dependencies {
  implementation("com.pambrose:prometheus-proxy:4.0.1")
}
```

## Example Configurations

- Basic setup: `examples/simple.conf`
- Multiple apps: `examples/myapps.conf`
- Metric filtering: `examples/agent-filters.conf`
- Dynamic discovery targets (watched file, not passed to `--config`): `examples/discovery-targets.conf`
- TLS (no mutual auth): `examples/tls-no-mutual-auth.conf`
- TLS (with mutual auth): `examples/tls-with-mutual-auth.conf`
- Prometheus federation: `examples/federate.conf`

## License

Apache License 2.0