Skip to content

[client] Add local Prometheus metrics endpoint#6689

Open
lixmal wants to merge 5 commits into
mainfrom
client-local-metrics
Open

[client] Add local Prometheus metrics endpoint#6689
lixmal wants to merge 5 commits into
mainfrom
client-local-metrics

Conversation

@lixmal

@lixmal lixmal commented Jul 7, 2026

Copy link
Copy Markdown
Collaborator

Adds an opt-in Prometheus /metrics endpoint to the client daemon so deployments can scrape connection health locally (requested for a RedHat evaluation: latency, connected peers, P2P vs relay).

  • Add netbird up --enable-local-metrics and --local-metrics-address (default 127.0.0.1:9191); the setting persists in the profile config and travels through SetConfig/Login like other config flags
  • Expose the existing client metrics (peer connection stage durations, sync and login durations) by mirroring them into a Prometheus registry behind the same recording interface
  • Add scrape-time gauges for current state: management/signal connectivity, known peers, connected peers by connection type, and per-peer latency for direct connections
  • Add a Grafana dashboard for the client next to the existing management/signal/relay dashboards
  • Gate the new settings behind --disable-update-settings and MDM policy (enableLocalMetrics, localMetricsAddress keys)
  • Regenerate the daemon JSON gateway, which was missing routes for recently added RPCs

All labels are bounded enums except the per-peer latency series, which carries the peer FQDN only.

Issue ticket number and link

Stack

Checklist

  • Is it a bug fix
  • Is a typo/documentation fix
  • Is a feature enhancement
  • It is a refactor
  • Created tests that fail without the change (if possible)
  • This change does not modify the public API, gRPC protocols, functionality behavior, CLI / service flags, or introduce a new feature — OR I have discussed it with the NetBird team beforehand (link the issue / Slack thread in the description). See CONTRIBUTING.md.

By submitting this pull request, you confirm that you have read and agree to the terms of the Contributor License Agreement.

Documentation

Select exactly one:

  • I added/updated documentation for this change
  • Documentation is not needed for this change (PoC under team discussion; docs to follow once the flag surface is finalized)

Docs PR URL (required if "docs added" is checked)

Paste the PR link from https://github.com/netbirdio/docs here:

https://github.com/netbirdio/docs/pull/__

Summary by CodeRabbit

  • New Features
    • Added a local Prometheus /metrics endpoint for the client, configurable via CLI and settings (enable/disable and listen address).
    • Wired local-metrics settings through configuration updates, login flows, and server lifecycle, including MDM-managed overrides.
    • Added a Grafana “Netbird / Client” dashboard to visualize connection, peer, latency, sync, and login metrics.
  • Bug Fixes
    • Improved connection timing metric calculations for more consistent stage duration reporting.
  • Tests
    • Added coverage for collecting/serving local metrics, CLI/config mapping, and MDM scenarios (including rejection when conflicting).

@coderabbitai

coderabbitai Bot commented Jul 7, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 5a34c23f-3a66-49a8-8628-8099177d4ed6

📥 Commits

Reviewing files that changed from the base of the PR and between 6d9cb90 and 35e157d.

📒 Files selected for processing (1)
  • client/internal/netflow/store/event_aggregation_test.go

📝 Walkthrough

Walkthrough

This PR adds an optional local Prometheus /metrics endpoint to the client. It introduces local metrics collection and serving, threads the new enable/address settings through config, CLI, proto, and server wiring, and adds dashboard and dependency updates.

Changes

Local metrics feature

Layer / File(s) Summary
Metrics stage duration helpers and Prometheus wrapper
client/internal/metrics/metrics.go, client/internal/metrics/influxdb.go, client/internal/metrics/metrics_default.go, client/internal/metrics/prometheus.go
Adds duration helpers, refactors InfluxDB duration recording, and wraps client metrics with a Prometheus registry and gatherer.
Local metrics Manager and collector
client/internal/localmetrics/localmetrics.go, client/internal/localmetrics/localmetrics_test.go, client/internal/peer/status.go
Implements the local /metrics server, collector, status snapshot helper, and tests.
Config, proto, and debug fields
client/internal/profilemanager/config.go, client/proto/daemon.proto, client/internal/debug/debug.go, client/mdm/policy.go, client/internal/profilemanager/config_mdm_test.go
Adds local metrics settings to config structures, policy keys, RPC messages, debug output, and MDM config coverage.
CLI flags and request propagation
client/cmd/root.go, client/cmd/up.go
Adds CLI flags and propagates their values into config, login, and set-config requests.
Daemon server reconciliation of local metrics
client/server/server.go, client/server/mdm.go, client/server/setconfig_test.go, client/server/setconfig_mdm_test.go
Wires local metrics into server lifecycle/config updates and extends request/MDM test coverage.
Dependencies and Grafana dashboard
go.mod, infrastructure_files/observability/grafana/dashboards/client.json
Updates Prometheus-related dependencies and adds a dashboard for the new client metrics.

Estimated code review effort: 4 (Complex) | ~60 minutes

Sequence Diagram(s)

sequenceDiagram
  participant CLI as Client CLI
  participant Server as Daemon Server
  participant Manager as localmetrics.Manager
  participant Status as peer.Status
  participant Prometheus as Prometheus Scraper

  CLI->>Server: SetConfigRequest / LoginRequest
  Server->>Server: update config and reconcile local metrics
  Server->>Manager: Reconcile(enabled, address)
  Prometheus->>Manager: GET /metrics
  Manager->>Status: GetPeerStates()
  Manager-->>Prometheus: metrics response
Loading

Possibly related PRs

  • netbirdio/netbird#6374: The local-metrics config fields are wired into the MDM enforcement and conflict handling changes in client/server/mdm.go and client/internal/profilemanager/config.go.

Suggested reviewers: pappz, mlsmaycon

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Title check ✅ Passed The title is concise and accurately describes the main change: adding a local Prometheus metrics endpoint for the client.
Description check ✅ Passed The description matches the template and covers the main sections, but the issue ticket number/link section is left blank.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch client-local-metrics

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🧹 Nitpick comments (1)
client/internal/localmetrics/localmetrics.go (1)

60-63: 🔒 Security & Privacy | 🔵 Trivial

Consider warning when binding to a non-loopback address.

The endpoint exposes peer FQDNs, latency, and connectivity state without authentication. Default is loopback-only, but addr is fully user-controlled; a log warning (or explicit opt-in) when the host isn't loopback would help users avoid accidental exposure.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@client/internal/localmetrics/localmetrics.go` around lines 60 - 63, The local
metrics endpoint in Manager.Reconcile can bind to any user-supplied addr, which
may accidentally expose sensitive peer and connectivity data. Update Reconcile
to detect when addr is not loopback after defaulting to DefaultListenAddress,
and emit a clear warning (or require explicit opt-in) before starting the
listener. Keep the check near the address selection logic so the behavior is
obvious and easy to maintain.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@client/internal/localmetrics/localmetrics.go`:
- Line 94: The local metrics server created in `Start` uses `http.Server`
without any timeout protections, leaving it vulnerable to slow client
connections. Update the `srv := &http.Server{...}` construction to set
appropriate `ReadTimeout`, `ReadHeaderTimeout`, `WriteTimeout`, and
`IdleTimeout` values, keeping the existing `Addr` and `Handler` behavior intact.
Use the `Start` function and the `http.Server` setup as the place to apply these
limits.

---

Nitpick comments:
In `@client/internal/localmetrics/localmetrics.go`:
- Around line 60-63: The local metrics endpoint in Manager.Reconcile can bind to
any user-supplied addr, which may accidentally expose sensitive peer and
connectivity data. Update Reconcile to detect when addr is not loopback after
defaulting to DefaultListenAddress, and emit a clear warning (or require
explicit opt-in) before starting the listener. Keep the check near the address
selection logic so the behavior is obvious and easy to maintain.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 7f956574-47e5-405e-a5e6-4581c2baf2ed

📥 Commits

Reviewing files that changed from the base of the PR and between d0d6dd4 and c4cfd8c.

⛔ Files ignored due to path filters (1)
  • client/proto/daemon.pb.go is excluded by !**/*.pb.go
📒 Files selected for processing (15)
  • client/cmd/root.go
  • client/cmd/up.go
  • client/internal/localmetrics/localmetrics.go
  • client/internal/localmetrics/localmetrics_test.go
  • client/internal/metrics/influxdb.go
  • client/internal/metrics/metrics.go
  • client/internal/metrics/metrics_default.go
  • client/internal/metrics/prometheus.go
  • client/internal/peer/status.go
  • client/internal/profilemanager/config.go
  • client/proto/daemon.proto
  • client/server/server.go
  • client/server/setconfig_test.go
  • go.mod
  • infrastructure_files/observability/grafana/dashboards/client.json

Comment thread client/internal/localmetrics/localmetrics.go Outdated
@github-actions

github-actions Bot commented Jul 7, 2026

Copy link
Copy Markdown

Release artifacts

Built for PR head 35e157d in workflow run #16543.

Artifact Link
All release artifacts Download
Linux packages Download
Windows packages Download
macOS packages Download
UI artifacts Download
UI macOS artifacts Download

GHCR images (amd64)

This comment is updated by the Release workflow. Artifact links expire according to the workflow retention policy.

@lixmal lixmal force-pushed the client-local-metrics branch from cd0ecae to 6d9cb90 Compare July 7, 2026 17:23
@sonarqubecloud

sonarqubecloud Bot commented Jul 8, 2026

Copy link
Copy Markdown

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant