Decide whether messaging receive telemetry should default on under stable semconv

[trask]

Is your feature request related to a problem? Please describe.

The v1.43 messaging semconv migration has introduced an inconsistency in whether receive telemetry is emitted by default when the user hasn't configured it explicitly. Raised by @laurit in #19351 (comment).

In #19351, KafkaInstrumenterFactory and KafkaTelemetryBuilder now track whether the user configured receive instrumentation at all, and fall back to the semconv mode when they didn't:

private boolean receiveInstrumentationEnabled() {
  return messagingReceiveInstrumentationConfigured
      ? messagingReceiveInstrumentationEnabled
      : emitStableMessagingSemconv();
}

So under stable semconv, Kafka receive telemetry defaults on. The sibling AWS SQS PR #19350 does not do this — it only widens the span-links condition to emitStableMessagingSemconv() || messagingReceiveInstrumentationEnabled, leaving the receive default off. Two instrumentations in the same migration stack now behave differently on the same question.

Describe the solution you'd like

Pick one default and apply it consistently across all messaging instrumentations:

1. Receive telemetry on by default under stable semconv. Stable semconv models receive as a first-class operation, and consumer-side context propagation is more legible with it. Costs an extra span per poll, including empty polls, which is why it was opt-in historically.
2. Keep it off by default regardless of semconv mode, preserving today's behavior; users opt in explicitly. This is what Migrate AWS SQS messaging telemetry to v1.43 #19350 does.

Whichever we choose, the messaging-receive-telemetry config option and its docs should state the default per semconv mode, and the Kafka and AWS SDK paths should agree.

Describe alternatives you've considered

Leaving each instrumentation to decide. Rejected: the default is user-facing and cross-cutting, and silently differing per instrumentation is worse than either uniform choice.

Additional context

Affected so far: kafka-clients-common-0.11 (KafkaInstrumenterFactory), kafka-clients-2.6 (KafkaTelemetryBuilder), aws-sdk-1.11 / aws-sdk-2.2 SQS. The v1.43 stack is #19350 → #19351 → #19352 and onward through #19357, so settling this soon avoids relitigating it in each one.

Tip

_{React with 👍 to help prioritize this issue. Please use comments to provide useful context, avoiding +1 or me too, to help us triage it. Learn more here.}

[trask]

Consolidating #19414 into this issue so one tracker owns the receive-default decision and implementation. #19414 adds one lifecycle question that should remain part of this work: in 2.x, the experimental receive-telemetry setting must continue to support legacy opt-in while acting as an explicit true/false override under stable messaging semconv; for 3.0, decide whether that setting should be retained/promoted or removed once stable pull receives default on. The implementation target is legacy unset off, stable/v3-preview unset on for true pull APIs, explicit true/false winning, and push/listener paths remaining process-only.

[laurit]

Semantic conventions state that

“Receive” spans SHOULD be created for operations of passing messages to the application when those operations are initiated by the application code (pull-based scenarios).

My interpretation of this is that Receive span should not be create by default for listener based instrumentations e.g. spring-kafka. This aligns with what we do now in default configuration with receive telemetry disabled. I guess one could argue that spring-kafka is built on kafka so there must be a call to poll somewhere, but based on that logic we could probably say that all push-based scenarios are really built on pull-based scenarios.
On the other hand this also reads that Receive should be created by regular kafka instrumentation since poll is called. This does not align what we do currently since we skip creating the receive span. Arguably the process spans that we create are more useful, but our instrumentation does not guarantee that they will be created. Semconv allows creating Process spans in addition to Receive spans.

“Process” spans MAY be created in addition to “Receive” spans for pull-based scenarios for operations of processing messages. Such spans could be created by application code, or by abstraction layers built on top of messaging SDKs.

Based on the above I get the feeling that semconv does not allow us to simply choose whether to enable or disable receive spans, but rather encourages us to choose the behavior based on whether the instrumented library is push or pull based.
Note that even now the behavior of the receive telemetry flag is ambiguous, some receive spans with span links are create even when receive telemetry is disabled. This is used in case where process spans are not created at all e.g. pull based instrumentations that don't create process spans.

[trask]

Settled where this lands. We're narrowing the receive telemetry we add for 3.0, and the policy for messaging receive is:

Mode	Receive spans	Receive metrics
legacy	unchanged (off unless explicitly enabled)	unchanged (none)
stable messaging semconv / v3-preview	off by default, opt-in	always recorded

When the opt-in is on, a receive span is created only when spanEligible = receiveSpansEnabled && (applicationInitiated || receivedMessageCount > 0). So an application-initiated pull produces a span even for a successful empty pull, while an internal listener polling loop produces one only for a non-empty poll — idle polling stays quiet.

@laurit, your point about whether a raw Kafka poll() counts as application-initiated is still open, but it no longer blocks this. Because receive spans are off by default in every mode now, nobody gets them unless they explicitly opt in. The application-initiated classification only affects opted-in users, so we can refine it later without a breaking default change.

Metrics are independent of that classification. Under stable/v3, messaging.client.operation.duration and friends are recorded for every pull, including empty pulls (duration with zero consumed messages), whether or not a span is created. Decoupling metrics from spans meant a new internal path that runs the full Instrumenter pipeline — attribute extractors, operation/metrics listeners, context customizers — while skipping span creation (#19548).

Because the setting now controls span creation only, it's renamed from otel.instrumentation.messaging.experimental.receive-telemetry.enabled to otel.instrumentation.messaging.experimental.receive-spans.enabled (declarative key receive_spans/development), with the old names kept as deprecated aliases (#19549). The messaging semconv permits but does not require a span for an empty receive, so defaulting spans off is conformant. The rewrite lives in #19490.

[laurit]

receive spans are off by default in every mode

Not sure this is the best. Currently we have instrumentations that create receive spans even when the receive-telemetry flag is false e.g. pulsar https://github.com/open-telemetry/opentelemetry-java-instrumentation/blob/c825b876d2cb42f693f434d5d0ab09ceb2beec1a/instrumentation/pulsar/pulsar-2.8/javaagent/src/main/java/io/opentelemetry/javaagent/instrumentation/pulsar/v2_8/telemetry/PulsarSingletons.java#L107-127 since processing spans are not created when using consumer.receive() we'd end up with nothing on the receiver side.

[trask]

Thanks @laurit, here is the end state I would like to get to.

Messaging receive telemetry

A receive span represents the application asking the broker for messages. It is created whenever the
application calls a pull API, and it is not created when a framework or the client library polls on
the application's behalf to feed a listener.

otel.instrumentation.messaging.experimental.receive-telemetry.enabled, or
java.common.messaging.receive_telemetry/development.enabled in declarative config, defaults to
false and opts into instrumenting that internal polling as well. It does not affect application
pulls.

Instrumentation	Application calls the pull API	Application registers a listener
Kafka	receive span from poll(), plus a process span per record as the application iterates	process span per record, plus a receive span from the framework's poll when true
AWS SQS	receive span from receiveMessage(), plus a process span per message as the application iterates	process span per message, plus a receive span from the framework's poll when true
Pulsar	receive span from consumer.receive(), no process span	process span from the listener, plus a receive span when true
JMS	receive span from MessageConsumer.receive(), no process span	process span from the listener, plus a receive span when true
RabbitMQ	receive span from basicGet(), no process span	process span from the pushed delivery, nothing polled to instrument
RocketMQ	receive span from SimpleConsumer.receive(), no process span	process span from the listener, plus a receive span when true

Kafka and AWS SQS have no listener API in the client, so their process spans are synthesized: the
instrumentation wraps the returned Iterable<ConsumerRecord>, or the SQS message list, and starts a
process span as the application iterates it.

Empty and failed receives

A receive that returns no messages creates no span, because an idle poll loop would otherwise produce
spans without limit. It still records metrics, so that the count of receive operations remains a
usable denominator for error rate; messaging.client.consumed.messages simply does not increment.
This does put the caller's poll timeout into the duration histogram.

A receive that fails creates a span and records metrics, with error.type set.

Legacy versus stable

Under stable conventions the producer is always a span link. Whether it is also the parent depends on
the ambient context, not on this setting.

Under legacy the setting decides. With it off, the producer parents the consumer span. With it on,
the producer is linked instead.

messaging.client.consumed.messages is recorded once per delivery under stable conventions, by the
receive operation where one exists and by the process operation otherwise.

Library instrumentation

Library instrumentation does not read the configuration property. Where a builder exposes an
equivalent setter, such as KafkaTelemetryBuilder.setMessagingReceiveTelemetryEnabled(boolean), it
defaults to false and must be set explicitly.

Getting there is a handful of PRs, since several instrumentations differ from this today. I will
track those separately.

[trask]

Plan: getting from the current PRs to the end state

The end state is documented in the #19433 comment. This is the complete gap list, with a difficulty
rating for each item and where it belongs.

The current PR set merges with only the two small changes in the first table. Everything else,
including the two large items, follows as separate PRs afterwards.

Fold into the existing PRs

#	Change	Where	Rating
1	Guard the empty basicGet() so it creates no receive span	#19500, or a small standalone PR	Simple. One condition, throwable == null && response == null, at RabbitChannelInstrumentation:427. Pure bug fix, matches every other instrumentation.
2	Ungate the basicGet() receive span so a pull always produces one	#19500	Simple in code, notable in behavior. Drop .setEnabled(...) at RabbitSingletons:178. Verified isolated: receiveInstrumenter() is used only by the basicGet path, deliverInstrumenter() covers push. It does add a span by default where there is none today.

Nothing else is small enough to fold in without turning a review of messaging metrics into a review
of receive-span policy.

Failed receives, one PR each

#	Change	Rating
3	Pulsar failed receive	Moderate. See closed #19570 for batches. Single needs a nullable PulsarRequest.message, which ripples into PulsarMessagingAttributesGetter for message id and payload size.
4	Kafka failed receive	Moderate. PollAdvice returns on empty before reading error. Recording a failure needs a KafkaReceiveRequest with no records, and the destination derives from records.partitions(), so it would have none.
5	JMS failed receive	Moderate. if (message == null) return; runs first. Needs a MessageWithDestination with no message, rippling into JmsMessageAttributesGetter.
6	AWS SQS failed receive	Larger. Failures never reach the receive instrumentation at all: SqsAccess.afterReceiveMessageExecution is called only from afterExecution, never onExecutionFailure. Needs new wiring, not a guard change.

Empty receives record metrics

Today every instrumentation returns early before the instrumenter starts, so the span and the
metrics are suppressed together. The end state keeps the span suppressed and keeps the metrics, so
that the operation count stays a usable denominator for error rate.

#	Change	Rating
7	Reland an API that records an operation without a span	Moderate, and it needs API review. This is exactly what the closed #19548 provided: startAndEndWithoutSpan across Instrumenter, InstrumenterAccess and InstrumenterUtil, six files with Instrumenter at +68/−19. The branch survives as archive/19548-record-without-span at e9bb378fe3, so it can be relanded rather than rewritten. It lands in instrumentation-api, so it draws wider review than a single instrumentation would.
8	Use it in each instrumentation	Moderate, five modules. Kafka, Pulsar, AWS SQS and JMS each convert their early return into a record-without-span call. RabbitMQ needs it too, because after item 1 its empty basicGet() records nothing either.

Item 8 depends on item 7. Neither depends on anything else.

Listener-path suppression, and ungating the Kafka and AWS SQS pull APIs

These are one change in two halves. Suppressing receive spans under a listener is what makes it safe
to stop gating the pull API, and ungating the pull API is what makes the suppression worth having.
Shipping either alone is wrong: suppression alone leaves pull APIs gated for no reason, and ungating
alone emits a receive span for every idle framework poll, roughly 864k per day for a consumer
polling at 100 ms.

#	Change	Rating
9	Introduce the marker and wire the four Kafka frameworks that already carry the signal	Moderate. Spring for Apache Kafka, Kafka Streams, Kafka Connect and the Vert.x client each already disable per-record wrapping around their poll loop, and PollAdvice captures that as previousValue on the line before it decides whether to start a receive span. Prefer a dedicated marker over reusing wrappingEnabled, which currently means "do not create per-record process spans" and would otherwise couple two decisions that only happen to coincide.
10	Reactor Kafka marker	Small. It disables wrapping only around record delivery in TracingDisablingKafkaFlux.onNext, so it looks like an application pull today. It already instruments reactor.kafka.receiver.internals.ConsumerHandler, which owns the consumer, so there is a natural place to mark the poll.
11	Spring Cloud AWS marker for SQS	Larger. No equivalent signal exists at all, so this is a new mechanism rather than a rewiring.
12	Stop gating the Kafka and AWS SQS pull-API receive spans	Small on its own, but gated by the above. Ships with 9 to 11, not before.