otel: add to main

[jazzlyn]

Summary

Deploy the OpenTelemetry-native observability foundation in the main cluster. Replace deprecated Promtail with the OTel Collector's filelog receiver, add OTLP ingress for external sources (opencode, node_exporter), and add Tempo as the tracing backend. Zero disruption to existing Prometheus ServiceMonitors, Loki, or Grafana.

Architecture

flowchart TD
    subgraph external["External Sources"]
        opencode["local opencode"]
        nodeexp["local node_exporter"]
    end

    subgraph main["Main Cluster - observability namespace"]
        subgraph collector["OTel Collector DaemonSet"]
            recv["Receivers:<br/>otlp (gRPC :4317, HTTP :4318)<br/>filelog (/var/log/pods)"]
            proc["Processors:<br/>batch, memory_limiter, k8sattributes"]
            exp["Exporters:<br/>prometheusremotewrite<br/>otlphttp/loki<br/>otlp/tempo"]
        end

        prom["Prometheus<br/><i>ServiceMonitors unchanged</i>"]
        loki["Loki 3.x<br/><i>OTLP endpoint enabled</i>"]
        tempo["Tempo<br/><i>SingleBinary, S3-backed</i>"]
        grafana["Grafana"]
        promtail["Promtail<br/><i>❌ removed</i>"]
    end

    opencode -->|"OTLP HTTP<br/>HTTPRoute + bearer token"| collector
    nodeexp -->|"OTLP HTTP<br/>HTTPRoute + bearer token"| collector
    collector -->|prometheusremotewrite| prom
    collector -->|otlphttp/loki| loki
    collector -->|otlp/tempo| tempo
    prom --> grafana
    loki --> grafana
    tempo --> grafana
    promtail -.->|"replaced by filelog"| collector

Loading

Decisions

• opentelemetry-operator chart via OCIRepository (CRD-based, auto-instrumentation for future)
• OTel Collector as DaemonSet CR (filelog replaces Promtail; otlp receiver handles external + future app pushes)
• Tempo SingleBinary, S3-backed (same MinIO as Loki), metrics generator → Prometheus
• Loki OTLP /otlp/v1/logs already available in 3.7.6 — just needs limits_config.otlp_config
• External access via spec.httpRoute (native CRD) with bearer token auth from OpenBao
• Existing Prometheus scraping completely untouched (~40 ServiceMonitors continue as-is)

Files

• New: opentelemetry/ (flux-sync, operator/, collector/, observability/) — 9 files
• New: tempo/ (flux-sync, app/, observability/) — 7 files
• Modified: parent kustomization.yaml, datasource.yaml (add Tempo), loki/helm-release.yaml (otlp_config)
• Removed: promtail/ directory + parent reference

Spec

https://github.com/tyriis/home-ops/blob/feature/otel-observability/docs/superpowers/specs/2026-08-12-otel-observability-design.md

[jazzlyn]

the existing observability setup will stay for now.

in phase 1 setup for otel operator and collector will be done alongside the existing stack. tempo will be added for traces. external sources can be connected to otel.