Skip to content

database.{NAME}.* telemetry is keyed by plugin type, not by mount or connection name (docs inaccuracy + request for per-mount labels) #32077

Description

@vinanrra

Summary

Two related things:

  1. Docs bugDatabase telemetry metrics describes the {NAME} in database.{NAME}.* as the named secrets engine. It is actually the database plugin type string, and for PostgreSQL it isn't even postgresql — it's pgx.
  2. Feature request — there is currently no way to break database secrets engine telemetry down per mount or per configured connection. I'd like that.

The documentation is inaccurate

The page currently states that enabling a PostgreSQL secrets engine called postgresql-prod produces database.postgresql-prod.CreateUser.error, and describes database.{NAME}.Close as the time to close "the database secrets engine {NAME}".

Both are wrong. {NAME} is neither the mount path nor the connection name, and the PostgreSQL plugin's type string is pgx. The real metric is:

database.pgx.CreateUser.error

Where it comes from

{NAME} is typeStr, obtained from the plugin itself in sdk/database/dbplugin/v5/plugin_factory.go (~L87):

typeStr, err := db.Type()
...
// Wrap with metrics middleware
db = &databaseMetricsMiddleware{
	next:    db,
	typeStr: typeStr,
}

sdk/database/dbplugin/v5/middleware.go emits it directly, with no labels:

metrics.MeasureSince([]string{"database", mw.typeStr, "Close"}, now)

The connection name never reaches the factory. In builtin/logical/database/backend_ce.go the connection name (name) is in scope and used for the connection cache, but only config.PluginName is forwarded:

dbw, err := newDatabaseWrapper(ctx, config.PluginName, pluginVersion, b.System(), b.logger)

Reproduction

I confirmed this by running the engine with an in-memory telemetry sink: mounted the database secrets engine, registered the postgresql-database-plugin, and created a connection named zzz-my-distinctive-connection-name. Every metric emitted:

counter  database.Close
counter  database.Initialize
counter  database.pgx.Close
counter  database.pgx.Initialize
sample   database.Close
sample   database.Initialize
sample   database.pgx.Close
sample   database.pgx.Initialize

key contains connection name "zzz-my-distinctive-connection-name" : false
key contains "pgx"                                                : true
key contains "postgresql-database-plugin"                          : false

Vault's own debug log in that run:

[DEBUG] got database plugin instance: type=pgx
[DEBUG] created database object: name=zzz-my-distinctive-connection-name plugin_name=postgresql-database-plugin

The connection name is known to Vault at that exact moment and still does not appear.

Actual {NAME} values

Verified by executing each plugin's Type():

Plugin {NAME}
postgresql-database-plugin pgx
mysql-database-plugin mysql
mssql-database-plugin mssql
mongodb-database-plugin mongodb
cassandra-database-plugin cassandra

Read from source constants but not executed: hanahdb, influxdbinfluxdb, redshiftredshift. The redshift plugin's constant carries a comment saying it exists specifically for how the plugin appears in metrics middleware.

Practical consequence

Every PostgreSQL connection, across every mount and every namespace, aggregates into a single database.pgx.* series. Operators reading the current docs will build dashboards expecting per-mount series that will never exist.

Feature request — per-mount and per-connection labels

Problem

I want to answer questions like "which mount is generating credential churn" and "is orders-primary slower than reporting-replica". Today that's impossible: the only dimension available is plugin type.

Proposed solution

Attach mount_point and connection_name as labels, gated behind a new telemetry option, off by default:

telemetry {
  add_mount_point_database_metrics = true
}

Resulting emission:

# default (option absent or false) — unchanged from today
database.Initialize
database.pgx.Initialize

# with the option enabled
database.Initialize;mount_point=db-prod/;connection_name=orders-primary
database.pgx.Initialize;mount_point=db-prod/;connection_name=orders-primary

Why labels rather than name segments

vault.rollback.attempt.{MOUNT_POINT} and vault.route.rollback.{MOUNT_POINT} were deliberately replaced with unsuffixed names because mount points in metric names explode Prometheus series counts, with add_mount_point_rollback_metrics added as the opt-in escape hatch. Repeating that pattern here would recreate the same problem and break existing queries. Labels leave metric names untouched, so current dashboards keep working and aggregate exactly as before.

Why opt-in

Series count scales with mounts × connections × operations. Both closest precedents (add_mount_point_rollback_metrics, add_lease_metrics_namespace_labels) are booleans defaulting to false, so this follows the established convention.

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions