Description of the bug
Reading our traces from the system we're migrating off of, it does seem like maybe it's just treating a trailing semicolon or some kind of empty-exec call as its own statement. That said, I wrote some tests that run multiple statements with and without semicolons and they don't produce this.
Share details about your runtime
Operating system details: Linux, Various, MacOS
RUBY_ENGINE: "ruby"
RUBY_VERSION: "3.4.8"
RUBY_DESCRIPTION: "ruby 3.4.8 (2025-12-17 revision 995b59f666) +PRISM [arm64-darwin24]"
Share a simplified reproduction if possible
I cannot reproduce as the very system I'd need to reproduce the inputs is the one failing to show me what is happening. I will share my otel SDK initializer though.
# typed: true
# frozen_string_literal: true
# OpenTelemetry Metrics Configuration
# This initializer sets up the OpenTelemetry SDK for user-facing metrics export.
# Metrics are sent to an OTLP collector endpoint. Operational metrics are in opentelemetry_traces.rb.
if Rails.env.test? || ENV["CI"]
Settings.opentelemetry.custom_traces_enabled= true
end
return unless Settings.opentelemetry.enabled || Settings.opentelemetry.traces_enabled
# Configure delta temporality for counters and histograms.
# UpDownCounter hardcodes :cumulative in its default_aggregation and is unaffected.
ENV['OTEL_EXPORTER_OTLP_METRICS_TEMPORALITY_PREFERENCE'] = 'delta'
# Prevent SDK auto-configuration from creating implicit default OTLP exporters.
# Atlas is metrics-only; the SDK defaults OTEL_TRACES_EXPORTER to 'otlp' when the
# opentelemetry-exporter-otlp gem is loaded, which would spin up an unauthenticated
# BatchSpanProcessor background thread pointing at OTEL_EXPORTER_OTLP_ENDPOINT.
ENV["OTEL_METRICS_EXPORTER"] ||= "none"
ENV["OTEL_TRACES_EXPORTER"] ||= "none"
require "opentelemetry/sdk"
require "opentelemetry/exporter/otlp"
require "opentelemetry/exporter/otlp_metrics"
require "opentelemetry/resource/detector"
require "opentelemetry-metrics-sdk"
require "open_telemetry/datadog_logging_metrics_exporter"
require "open_telemetry/filtering_span_processor"
# Note: When disabled, the OTel gems are never loaded. This is safe because the
# OpenTelemetryMetrics module guards all code paths with enabled? checks, which
# call settings_enabled? before touching any OpenTelemetry SDK classes.
if Settings.opentelemetry.traces_enabled
require "opentelemetry/instrumentation/rack"
require "opentelemetry/instrumentation/rails"
end
# AuthenticatedMetricsExporter wraps the OTLP MetricsExporter to support dynamic
# bearer token authentication via Okta OAuth2, matching the pattern used by
# HCP billing (OktaOAuth2Config) and HCP log service (AuthTokens::LogService).
#
# The standard OTLP exporter only accepts static headers at construction time.
# This subclass refreshes the Authorization header before each export call,
# ensuring the bearer token is always current even across token rotation.
#
# When Okta credentials are not configured, it falls back to static headers
# from OTEL_EXPORTER_OTLP_HEADERS (e.g. for local development or testing).
class AuthenticatedMetricsExporter < OpenTelemetry::DatadogLoggingMetricsExporter
def initialize(token_provider:, **kwargs)
super(**kwargs)
@token_provider = token_provider
end
def before_export
refresh_authorization_header
end
private
def refresh_authorization_header
token = @token_provider.call
@headers["Authorization"] = "Bearer #{token}" if token.present?
rescue StandardError => e
Rails.logger.warn("[OpenTelemetry] Failed to refresh auth token: #{e.class} - #{e.message}")
end
end
# Force OTLP trace export to bypass Atlas default proxying by explicitly
# setting Net::HTTP proxy address to nil. This should eventually be pushed upstream
# via this issue: https://github.com/open-telemetry/opentelemetry-ruby/issues/2166
class DirectHttpProxyTraceExporter < OpenTelemetry::Exporter::OTLP::Exporter
private
KEEP_ALIVE_TIMEOUT = 30 # seconds
def http_connection(uri, ssl_verify_mode, certificate_file, client_certificate_file, client_key_file)
http = Net::HTTP.new(uri.hostname, uri.port, nil)
http.use_ssl = uri.scheme == "https"
http.verify_mode = ssl_verify_mode
http.ca_file = certificate_file unless certificate_file.nil?
http.cert = OpenSSL::X509::Certificate.new(File.read(client_certificate_file)) unless client_certificate_file.nil?
http.key = OpenSSL::PKey::RSA.new(File.read(client_key_file)) unless client_key_file.nil?
http.keep_alive_timeout = KEEP_ALIVE_TIMEOUT
http
end
end
metrics_endpoint = if Settings.opentelemetry.otlp_endpoint.present?
# Create the OTLP metrics exporter
# Default compression is gzip, timeout is 10 seconds
# The endpoint must include the /v1/metrics path for HTTP protocol
raw_endpoint = Settings.opentelemetry.otlp_endpoint
raw_endpoint.end_with?("/v1/metrics") ? raw_endpoint : "#{raw_endpoint}/v1/metrics"
end
traces_endpoint = if Settings.opentelemetry.traces_otlp_endpoint.present?
# Create the OTLP trace exporter
# Default compression is gzip, timeout is 10 seconds
# The endpoint must include the /v1/traces path for HTTP protocol
raw_endpoint = Settings.opentelemetry.traces_otlp_endpoint
raw_endpoint.end_with?("/v1/traces") ? raw_endpoint : "#{raw_endpoint}/v1/traces"
end
# Build headers from static configuration (OTEL_EXPORTER_OTLP_HEADERS).
# These are used as the base headers; dynamic auth will override the Authorization header.
parsed_headers = {}
if Settings.opentelemetry.otlp_headers.present?
Settings.opentelemetry.otlp_headers.split(",").each do |header|
key, value = header.split("=", 2)
parsed_headers[key.strip] = value&.strip if key.present?
end
end
metrics_exporter_options = {
endpoint: metrics_endpoint,
}
metrics_exporter_options[:headers] = parsed_headers if parsed_headers.any?
trace_exporter_options = {
endpoint: traces_endpoint,
}
trace_exporter_options[:headers] = parsed_headers if parsed_headers.any?
# Use dynamic Okta token auth when credentials are configured (production/staging),
# otherwise fall back to static headers in development only.
okta_configured = Settings.opentelemetry.okta_client_id.present? &&
Settings.opentelemetry.okta_client_secret.present?
metrics_otlp_exporter = if metrics_endpoint.present?
if okta_configured
token_provider = -> { HCP::AuthTokens::OpenTelemetry.instance.access_token(min_ttl: 60) }
AuthenticatedMetricsExporter.new(token_provider: token_provider, **metrics_exporter_options)
elsif Rails.env.development?
OpenTelemetry::DatadogLoggingMetricsExporter.new(**metrics_exporter_options)
else
Rails.logger.warn("[OpenTelemetry] Okta credentials required in non-development environments")
nil
end
end
# Wrap in PeriodicMetricReader for automatic background export
periodic_metric_reader = if metrics_otlp_exporter
OpenTelemetry::SDK::Metrics::Export::PeriodicMetricReader.new(
exporter: metrics_otlp_exporter,
export_interval_millis: Settings.opentelemetry.export_interval * 1000,
export_timeout_millis: 30_000
)
end
span_processor = if traces_endpoint.present?
trace_exporter = DirectHttpProxyTraceExporter.new(**trace_exporter_options)
batch_span_processor = OpenTelemetry::SDK::Trace::Export::BatchSpanProcessor.new(trace_exporter)
OpenTelemetry::FilteringSpanProcessor.new(delegate: batch_span_processor)
end
# Configure the meter provider with our periodic reader
OpenTelemetry::SDK.configure do |c|
c.service_name = Settings.opentelemetry.service_name
c.service_version = TFP.release || "unknown"
c.resource = OpenTelemetry::SDK::Resources::Resource.create(
OpenTelemetry::SemanticConventions::Resource::SERVICE_INSTANCE_ID => "#{ENV.fetch("NOMAD_ALLOC_ID", SecureRandom.uuid)}.#{Process.pid}"
)
c.resource = OpenTelemetry::Resource::Detector::Container.detect
if span_processor && Settings.opentelemetry.traces_enabled
c.add_span_processor(span_processor)
c.logger = Rails.logger
c.use_all({ 'OpenTelemetry::Instrumentation::Sidekiq' => { span_naming: :job_class } })
end
end
# Add the metric reader to the meter provider
# Currently all of the metrics that use the meter_provider are customer-facing, but that may not be true
# in the future as we add OpenTelemeetry instrumentation.
# TODO: Make this filter to only customer-provided metrics that we explicitly emit via OpenTelemetryMetrics
OpenTelemetry.meter_provider.add_metric_reader(periodic_metric_reader) if metrics_otlp_exporter
# 'Tracer' can be used throughout your code now
OpenTelemetry.tracer_provider.tracer(Settings.opentelemetry.service_name)
# Register shutdown hook for graceful cleanup in non-Sidekiq processes (Puma).
# Sidekiq flushes metrics via OpenTelemetryMetrics.flush in its on(:quiet) hook
# (see config/initializers/sidekiq.rb). Do NOT trap SIGTERM here — it interferes
# with Sidekiq's graceful shutdown and causes SuperFetch job recoveries on deploy.
unless defined?(Sidekiq) && Sidekiq.server?
at_exit do
OpenTelemetry.meter_provider.shutdown
OpenTelemetry.tracer_provider.shutdown
rescue StandardError => e
Rails.logger.warn("[OpenTelemetry] Error during shutdown: #{e.message}")
end
end
Rails.logger.info("[OpenTelemetry] Metrics export enabled, endpoint: #{metrics_endpoint}") if metrics_otlp_exporter
Rails.logger.info("[OpenTelemetry] Traces export enabled, endpoint: #{traces_endpoint}") if span_processorTip: React with 👍 to help prioritize this issue. Please use comments to provide useful context, avoiding +1 or me too, to help us triage it. Learn more in our end user docs.
Description of the bug
Reading our traces from the system we're migrating off of, it does seem like maybe it's just treating a trailing semicolon or some kind of empty-exec call as its own statement. That said, I wrote some tests that run multiple statements with and without semicolons and they don't produce this.
Share details about your runtime
Operating system details: Linux, Various, MacOS
RUBY_ENGINE: "ruby"
RUBY_VERSION: "3.4.8"
RUBY_DESCRIPTION: "ruby 3.4.8 (2025-12-17 revision 995b59f666) +PRISM [arm64-darwin24]"
Share a simplified reproduction if possible
I cannot reproduce as the very system I'd need to reproduce the inputs is the one failing to show me what is happening. I will share my otel SDK initializer though.
Tip: React with 👍 to help prioritize this issue. Please use comments to provide useful context, avoiding
+1orme too, to help us triage it. Learn more in our end user docs.