Consider classifying context-scoped attributes so providers can filter them

[RKest]

Consider classifying context-scoped attributes so providers can filter them

The OTEP (#4931)
carries context-scoped attributes as a single bag that is stamped onto every signal the
feature is enabled for. Enablement is per-signal, but it is configured on the provider and
applies to the whole bag: a signal either receives all context-scoped attributes or none of
them. This issue proposes discussing whether attributes should be able to declare properties
of the data they carry, so that the existing per-signal opt-in can filter on those properties
instead of being all-or-nothing.

Motivation

Whether an attribute belongs on a given signal depends on two facts that are known in two
different places:

• What the data is — is this an end-user identifier? is its value unbounded? Known by the
  code that sets the attribute, which has the domain context.
• What a backend will accept — is this tracing backend approved to store personal data? can
  this metrics pipeline absorb an unbounded dimension? Known by whoever configures the SDK for
  a given deployment.

The OTEP gives the second party a switch and the first party nothing, and the switch is coarse.
Two cases it cannot express:

• Different backends have different data-handling rules. A logging backend may be
  approved to store user identifiers or other PII, while the tracing backend it sits next to
  is not — traces are widely readable, sampled independently, and often exported to third
  parties. An operator who wants user.id on log records and not on spans has no way to
  say so.
• High-cardinality attributes are fine on spans and logs, and fatal on metrics. A
  conversation, session, or request identifier is valuable on a span and on a log record, and
  is a cardinality explosion on a metric — every distinct value forks a time series. Today the
  only protection is to disable context-scoped attributes on the meter provider entirely,
  which also discards the low-cardinality attributes that were the reason to enable the
  feature.

The workaround available under the current design makes the problem concrete. To keep
user.id off spans, an operator disables context-scoped attributes on the tracer provider —
which also drops every other context-scoped attribute from spans, including the ones that
motivated using the feature. The gate is all-or-nothing per signal, so the two requirements
cannot be satisfied at once.

Neither party can solve this alone. A library that sets user.id cannot know whether a
particular deployment's tracing backend accepts personal data; that varies by operator,
jurisdiction, and vendor. An operator cannot know that a given attribute key holds personal
data without reading the source of every library in the process. So the setter should state a
fact about the data, and the provider should apply a policy to it.

Proposal

Let each group of attributes carry labels asserting properties of the data, and let the
per-signal provider configuration state which assertions it requires. Sketched in Python:

class ContextScopedAttributes(TypedDict):
    attributes: types.Attributes
    # Assertions about the data. Defaults to (), which asserts nothing.
    labels: NotRequired[tuple[Literal["low-cardinality", "non-pii"], ...]]


def add_context_scoped_attributes(
    attributes: tuple[ContextScopedAttributes, ...],
    context: context_api.Context | None = None,
) -> context_api.Context: ...

Labels are positive assertions, and the absence of one means unknown, not safe. An
attribute reaches a signal only if it carries every assertion that signal requires, so the
default of () is the conservative case: it flows only where nothing is required.

Call site — the setter states what it knows about the data and says nothing about signals:

add_context_scoped_attributes((
    {
        # Personal, and unbounded: nothing can be asserted.
        "attributes": {"user.id": "<user id>"},
    },
    {
        # Bounded set of agent names, not personal.
        "attributes": {"gen_ai.agent.name": "<agent name>"},
        "labels": ("low-cardinality", "non-pii"),
    },
    {
        # Not personal, but unbounded — no cardinality assertion.
        "attributes": {"gen_ai.conversation.id": "<conversation id>"},
        "labels": ("non-pii",),
    },
))

Configuration — the operator states what each backend requires, extending the OTEP's existing
per-signal switch rather than replacing it:

tracer_provider:
  context_scoped_attributes:
    enabled: true
    require: [non-pii]
logger_provider:
  context_scoped_attributes:
    enabled: true             # approved for personal data; requires no assertions
meter_provider:
  context_scoped_attributes:
    enabled: true
    require: [low-cardinality, non-pii]

The same three attributes then land differently on each signal, without any call site knowing
which backends are in use, and without the operator knowing which keys are sensitive:
gen_ai.agent.name reaches all three, gen_ai.conversation.id reaches spans and logs, and
user.id reaches logs only.

The failure mode is what motivates the direction. If labels named hazards instead
(pii, high-cardinality), then forgetting to label — a new attribute added by a library, an
existing one whose values become unbounded — would silently push that data into every backend,
including the ones that must not receive it. With assertions, forgetting to label costs some
telemetry. Missing telemetry is noticed and fixed; a leaked identifier or a cardinality
explosion is not undone.

Points to settle if the direction is accepted:

• What each signal requires by default. Requiring low-cardinality on the meter provider
  out of the box is the case with the least argument against it: cardinality damage is
  expensive, easy to cause by accident, and often noticed only once a backend starts dropping
  series. Whether the tracer provider should require non-pii by default is a harder call,
  since most deployments are not PII-restricted and would see attributes disappear until call
  sites are updated.
• Back-compatibility with the OTEP's current behaviour. Under 4931 today, enabling the
  feature stamps every context-scoped attribute onto every enabled signal. Any non-empty
  default require changes that: existing setters assert nothing, so their attributes stop
  reaching the signals that now require an assertion. That is the intended trade — fail closed
  rather than leak — but it is a behavioural change and should be called out as one rather than
  discovered.
• Naming. labels collides with prior art where "label" means what OpenTelemetry now
  calls an attribute (Prometheus, OpenCensus). classification, categories, or tags may
  read better in spec text.
• Whether filtering should also reach finer than a signal, e.g. a specific instrument or a
  specific span. Probably out of scope here, but it is the direction the same argument points
  in, and worth not designing against.

Alternatives Considered

• Provider-level per-signal opt-in only (the current design). All-or-nothing per signal,
  so any context that mixes a sensitive attribute with a useful one cannot be configured
  correctly. This is the case that motivates the issue.
• Let the setter target signals directly, e.g. targets: ("spans", "logs") instead of
  labels: ("non-pii",). Simpler and more direct, and it does express the requirement — but it
  puts the routing decision in the wrong place. The setter is frequently a library, which
  cannot know whether this deployment's tracing backend accepts personal data; it would have
  to hardcode a guess that is wrong for some users and unchangeable without editing the call
  site. Call sites would also need revisiting whenever a deployment's backends change.
  Classification keeps each party stating only what it actually knows.

Related use cases

• Multi-tenancy with user-level identifiers. The OTEP's primary motivation is stamping
  tenant information onto all telemetry. Once user-level identifiers are in the same bag,
  the PII asymmetry between backends appears immediately.
• GenAI and agentic frameworks. An agentic framework knows the agent, workflow, and
  conversation; the model-client instrumentation a layer below it emits the inference
  telemetry and cannot learn any of it. Context is the only channel between them, and the
  attributes involved split cleanly along the lines above: agent name is bounded and not
  personal, conversation id is not personal but unbounded, and end-user identifiers are
  neither.
  Discussed in
  open-telemetry/opentelemetry-python-genai#337,
  which was closed in favour of solving this in the specification first.
• Per-request debugging identifiers. Correlation identifiers that are wanted on spans and
  logs for a single request, and that would be actively harmful as metric dimensions.

_{Tip: React with 👍 to help prioritize this issue. Please use comments to provide useful context, avoiding +1 or me too, to help us triage it. Learn more here.}