Metrics server silently fails to start when the configured port is already in use

[stefanamaerz]

What happened? How can we reproduce this?

What happened:

If Tetragon is started with --metrics-server set to an address/port that's already occupied by another process, it logs "Starting metrics server" but never reports the bind error and never actually serves metrics. The main process continues on as if everything is fine.

This makes the failure hard to diagnose.

How to reproduce:

1. Start another listener on port 2112:
   python3 -m http.server 2112 --bind 0.0.0.0
2. Start Tetragon with metrics enabled on the same tcp/2112:
   sudo tetragon --metrics-server=:2112
3. Observe:

• Tetragon logs: level=info msg="Starting metrics server" addr=:2112
• Tetragon continues and reaches "Listening for events..."
• ss -tlnp | grep 2112 shows only the Python process listening
• curl http://127.0.0.1:2112/metrics returns the Python HTTP server's response, not Prometheus metrics
• No error or warning is logged by Tetragon

Code reference:
pkg/metricsconfig/root.go:

func EnableMetrics(address string) {
    reg := GetRegistry()
    logger.GetLogger().Info("Starting metrics server", "addr", address)
    http.Handle("/metrics", promhttp.HandlerFor(reg, promhttp.HandlerOpts{Registry: reg}))
    http.ListenAndServe(address, nil)
}

http.ListenAndServe runs in a goroutine (called via go metricsconfig.EnableMetrics() in cmd/tetragon/main.go) with its return value ignored entirely.

Environment:

• Tetragon version: v1.7.0
• Platform: bare metal / systemd (also affects any non-Kubernetes deployment that enables --metrics-server)
• Kernel: 6.17.13

Impact:
In production deployments managed by configuration management tools (e.g. Chef) that may restart Tetragon rapidly, a port conflict can leave the service running but without working metrics, leading to silent observability gaps.

Proposed fix:
I would like to propose the following, roughly in order of increasing scope, and would appreciate maintainer input on how far to take it:

1. Log the bind error. Capture http.ListenAndServe's return value and log it instead of discarding it. This alone makes the failure observable and requires no API changes.
2. Accept a context.Context. Thread ctx into EnableMetrics and switch from the package-level http.ListenAndServe to a dedicated *http.Server, so the metrics server can be gracefully stopped via Shutdown(ctx) on ctx.Done() (currently there's no shutdown path for this server at all).
3. Optional: bind synchronously. If we want startup to fail loudly on a bad/occupied address (rather than only logging after the fact), move net.Listen out of the goroutine and into the synchronous startup path, returning the bind error to the caller so main() can abort startup.

Personally I would think 1 and 2 above is sufficent, 3 seems extreme for a secondary metrics system.

Tetragon Version

v1.7.0

Kernel Version

6.17.13

Kubernetes Version

n/a we are not running in kubernetes, however this probably also affects kube deployments

Bugtool

No response

Relevant log output

Anything else?

Let me know if I'm understanding the problem correctly, happy to put up a PR!

[mtardy]

Proposed fix: I would like to propose the following, roughly in order of increasing scope, and would appreciate maintainer input on how far to take it:

1. Log the bind error. Capture http.ListenAndServe's return value and log it instead of discarding it. This alone makes the failure observable and requires no API changes.
2. Accept a context.Context. Thread ctx into EnableMetrics and switch from the package-level http.ListenAndServe to a dedicated *http.Server, so the metrics server can be gracefully stopped via Shutdown(ctx) on ctx.Done() (currently there's no shutdown path for this server at all).
3. Optional: bind synchronously. If we want startup to fail loudly on a bad/occupied address (rather than only logging after the fact), move net.Listen out of the goroutine and into the synchronous startup path, returning the bind error to the caller so main() can abort startup.

[...]

Relevant log output

Anything else?

Let me know if I'm understanding the problem correctly, happy to put up a PR!

Indeed the code is really overlooking everything rn

tetragon/pkg/metricsconfig/root.go

Lines 28 to 34 in 947eaed

	func EnableMetrics(address string) {
	reg := GetRegistry()
	logger.GetLogger().Info("Starting metrics server", "addr", address)
	http.Handle("/metrics", promhttp.HandlerFor(reg, promhttp.HandlerOpts{Registry: reg}))
	http.ListenAndServe(address, nil)
	}

I think if you can rework that function so that it actually returns an error if something went wrong that would be great (and what will go wrong is bind as you mentioned). Then we can decide what to do from main, from logging to stop the execution there. I'd be fine with 3, to bind synchronously and start the server on a different go routine from that function, I think it's okay.

If you want to open a PR that would be a pleasure! Thanks!