Self-hosted Theia v0.9.0: custom images, Grafana 13, immutable plugins, schema fixes - #1377
Open
Scoobed wants to merge 43 commits into
Open
Self-hosted Theia v0.9.0: custom images, Grafana 13, immutable plugins, schema fixes#1377Scoobed wants to merge 43 commits into
Scoobed wants to merge 43 commits into
Conversation
- Pin antrea.io/antrea to v2.6.1 via local replace directive - Update go.mod to go 1.25.0; dependencies bumped by go mod tidy (k8s v0.35, etcd v3.6.5, otel v1.40, etc.) golangci-lint: - Upgrade golangci-lint from v1.54.2 to v2.12.2 in Makefile - Migrate .golangci.yml to v2 config format (linters.settings, govet, issues.exclude-files, drop deprecated unused linter) Antrea API migration (removed APIs -> CrdV1beta1): - test/e2e/framework.go: DeleteANP, DeleteACNP, CleanACNPs, DeleteV1Alpha2CG/V1Alpha3CG/CleanCGs all updated to CrdV1beta1 - test/e2e/flowvisibility_test.go: six AddIngress/AddEgress calls updated to use ANNPRuleBuilder struct API Go 1.26 vet fixes: - anomalydetector/controller.go: Infof -> InfoS (no format directives) - networkpolicyrecommendation/controller.go: same fix - pkg/controller/util_test.go: remove unnecessary fmt.Sprintf k8s v0.35 WatchListClient feature gate (enabled by default): - Add TestMain to anomalydetector and networkpolicyrecommendation test packages to disable WatchListClient for fake CRD clients ClickHouse v1->v2 driver migration (double-registration panic): - clickhouse_test.go: replace v1 import with v2, update DSN assertions - test_utils.go: update expected DSN to v2 clickhouse:// format - plugins/clickhouse-monitor/main_test.go: update expected DSN Co-Authored-By: Oz <oz-agent@warp.dev>
…2.6.1 clickhouse-go v2 upgrade: - Bump github.com/ClickHouse/clickhouse-go/v2 from v2.35.0 to v2.46.0 - Remove github.com/ClickHouse/clickhouse-go v1 (no longer imported) - Run go mod tidy (picks up ch-go v0.71, brotli v1.2, lz4 v4.1.25, etc.) ClickHouse schema (egressNodeName): - Add migration 000006_antrea-v2.6.up.sql: ALTER TABLE flows/flows_local ADD COLUMN egressNodeName String (Antrea v2.6.1 inserts this field) - Add corresponding 000006_antrea-v2.6.down.sql e2e framework: - Add EgressNodeName field to ClickHouseFullRow struct - Add egressNodeName to insertQueryflowtable INSERT column list - Fix SetupClickHouseConnection to use v2 clickhouse:// DSN format instead of old tcp:// format Co-Authored-By: Oz <oz-agent@warp.dev>
- Update Antrea prerequisite from v1.7.0 to v2.6.0 in getting-started.md - Add 'Upgrading an Existing Deployment' section with helm upgrade and kubectl rollout restart commands for enabling recordContents.podLabels - Add 'Verifying Pod Labels in ClickHouse' section with clickhouse-client queries to check label capture (spot-check and coverage count) - Add hack/configure-flow-aggregator-pod-labels.sh helper script that: - Detects the Flow Aggregator ConfigMap automatically - Enables podLabels via helm upgrade or direct ConfigMap patch - Restarts the deployment and waits for rollout - Verifies label capture in ClickHouse with retry logic - Supports --verify-only flag for read-only inspection Co-Authored-By: Oz <oz-agent@warp.dev>
Go version: - build/images/deps/go-version: 1.25.0 → 1.26.0 (matches go.mod requirement) - Makefile docker-test-unit: hardcoded golang:1.21 → golang:$(GO_VERSION) Base images (all Docker Hub): - ubuntu:22.04 → ubuntu:26.04 (latest, April 2026 LTS) in clickhouse-monitor - ubuntu:23.04 → ubuntu:26.04 (EOL → latest LTS) in theia-manager - clickhouse/clickhouse-server:23.4 → 25.8 (latest stable) in clickhouse-server Vendoring: - go mod vendor: captures all deps including local Antrea v2.6.1 checkout so Docker builds are fully self-contained (no host path dependencies) - Dockerfiles updated to GOFLAGS=-mod=vendor for reproducible image builds Co-Authored-By: Oz <oz-agent@warp.dev>
- Add docker-images and pull-upstream-images Makefile targets - Rename all upstream pull targets to use theia- prefix - Skip spark-operator from pull-upstream-images (pending new source) - Update clickhouse-operator install bundle from 0.21.0 to 0.27.0 - Update all values.yaml image repos to ghcr.io/scoobed/theia-* - Fix stale projects.registry.vmware.com busybox ref in grafana template - Update grafana tag to 11.6.0 and zookeeper tag to 3.8.4 Co-Authored-By: Oz <oz-agent@warp.dev>
Package the theia Helm chart with index.yaml for use as a self-hosted Helm repo directly from the Git repository, without requiring Artifact Hub. Co-Authored-By: Oz <oz-agent@warp.dev>
- Update ZooKeeper from 3.8.4 to 3.9.5 (latest current release) - Remove -XX:+CMSParallelRemarkEnabled from JVMFLAGS (CMS flag removed in JDK 17+, and G1GC is already configured) Co-Authored-By: Oz <oz-agent@warp.dev>
Only chown data/log dirs and the config files we generate (zoo.cfg, log4j.properties, java.env), skip pre-existing root-owned files in /conf that the ZooKeeper 3.9.5 image ships with. Co-Authored-By: Oz <oz-agent@warp.dev>
- Update Chart.yaml version from 0.0.0 to 0.9.0 - Update appVersion from latest to 0.9.0 - Enable dynamic PVC provisioning for ClickHouse (8Gi) and ZooKeeper (5Gi) using standard storage class - Fix ZooKeeper FQDN in ClickHouseInstallation Co-Authored-By: Oz <oz-agent@warp.dev>
Add egressNodeName, appProtocolName, and httpVals columns to: - Base table creation script (create_table.sh) - Migration 000006_antrea-v2.6 (up and down) - E2e test framework (ClickHouseFullRow struct and INSERT query) These columns are required by Antrea Flow Aggregator v2.6+ and were causing 'No such column' errors on flow record insertion. Co-Authored-By: Oz <oz-agent@warp.dev>
- Add EffectiveVersion to server config (required by newer k8s apiserver) - Generate OpenAPI definitions via openapi-gen for CRD types - Add OpenAPI V2/V3 config using generated definitions - Export Scheme from apiserver package for OpenAPI namer - Implement SingularNameProvider on all REST storage types - Add openapi-gen step to codegen script These fixes resolve nil pointer panics and registration errors caused by the vendored k8s.io/apiserver requiring EffectiveVersion, OpenAPI definitions, and SingularNameProvider that were not present in the original codebase. Co-Authored-By: Oz <oz-agent@warp.dev>
- Bump @grafana/data, @grafana/runtime to 11.x - Bump React 16 -> 18, react-google-charts 4.x -> 5.x - Simplify field.values extraction (plain arrays in Grafana 10+) - Migrate tests from enzyme to @testing-library/react - Fix tsconfig ts-node moduleResolution for @grafana/tsconfig v2 - Exclude test files from webpack type checking - Update grafanaDependency to >=11.0.0 - Bump plugin version to 2.0.0 Co-Authored-By: Oz <oz-agent@warp.dev>
- Add built plugin files to build/charts/theia/plugins/ - Create ConfigMap template for bundled plugin binary data - Add init-plugins init container to copy plugin from ConfigMap - Mount plugin into Grafana's plugin directory - Remove Sankey download URL from installPlugins values The plugin is now loaded from a ConfigMap at deploy time instead of being downloaded from downloads.antrea.io at startup. Co-Authored-By: Oz <oz-agent@warp.dev>
Chord plugin: - Bump to Grafana 11.x, React 18, d3@7.1.0 - Simplify field.values extraction (plain arrays) - Remove enzyme, migrate tests to @testing-library/react - Bundle in Helm chart ConfigMap (no download needed) Dependency plugin: - Migrate from @grafana/toolkit to webpack build system - Bump to Grafana 11.x, React 18, mermaid v11 - Rewrite to use mermaid async render API - Fix React hooks ordering - Still downloaded at startup (too large for ConfigMap) ClickHouse datasource remains in installPlugins (binary plugin). Co-Authored-By: Oz <oz-agent@warp.dev>
Add GF_PLUGINS_ALLOW_LOADING_UNSIGNED_PLUGINS env var to Grafana deployment for the three Theia custom plugins. Required because Grafana 11 rejects unsigned plugins by default. Co-Authored-By: Oz <oz-agent@warp.dev>
Updated packaged chart includes: - Bundled Sankey and Chord plugins (ConfigMap) - Grafana 11.x compatibility - Unsigned plugin allow-list - All prior fixes (ZooKeeper, ClickHouse schema, theia-manager) Co-Authored-By: Oz <oz-agent@warp.dev>
Grafana 11 no longer provides d3 as a global. Removing 'd3' from the webpack externals list ensures d3 is bundled into the plugin instead of expecting it from the Grafana host environment. This fixes the 'd3.sankey is not a function' error in the Sankey plugin and similar d3 reference errors in the Chord plugin. Co-Authored-By: Oz <oz-agent@warp.dev>
Grafana loads plugins via AMD and only exposes 'react' as a module, not 'react/jsx-runtime'. React 18's automatic JSX transform emits imports from 'react/jsx-runtime' which causes 404 errors at runtime. Configure SWC to use runtime: 'classic' which emits React.createElement calls instead, compatible with Grafana's AMD module system. Also removes d3 from webpack externals since Grafana 11 no longer provides d3 globally. Co-Authored-By: Oz <oz-agent@warp.dev>
Major changes: - Rewrite Sankey plugin with d3-sankey (replaces react-google-charts which had runtime dependency on Google Charts CDN causing d3.sankey errors in Grafana's sandboxed module environment) - Create Dockerfile.grafana based on grafana/grafana:13.0.1 (latest) - Pre-install grafana-clickhouse-datasource in the image - Copy all 3 Theia plugins into the image at build time - Set GF_PLUGINS_ALLOW_LOADING_UNSIGNED_PLUGINS in the image - Add theia-grafana Makefile target - Remove plugin ConfigMap template and bundled plugin files from chart - Remove init-plugins init container from Grafana deployment - Remove installPlugins from values.yaml - Remove plugin volume mounts - Update image to ghcr.io/scoobed/theia-grafana:v0.9.0 The install is now fully immutable - no internet access needed at pod startup, no runtime plugin downloads, no ConfigMaps for plugins. Co-Authored-By: Oz <oz-agent@warp.dev>
The PVC mounted at /var/lib/grafana was hiding the plugins baked into the image at /var/lib/grafana/plugins/. Move all plugins to /opt/grafana/plugins and set GF_PATHS_PLUGINS accordingly. Also install clickhouse-datasource with --pluginsDir to the same custom path. All plugins now show correct v2.0.0 versions. Co-Authored-By: Oz <oz-agent@warp.dev>
Co-Authored-By: Oz <oz-agent@warp.dev>
- Add self-hosted installation section to getting-started.md with full prerequisites, install steps, image table, and build commands - Restructure doc to separate self-hosted (recommended) from upstream - Repackage Helm chart v0.9.0 with Grafana 13.0.2 custom image Co-Authored-By: Oz <oz-agent@warp.dev>
Key changes: - Migrate all Docker images to ghcr.io/scoobed with theia- prefix - Build custom theia-grafana image (Grafana 13.0.2) with all plugins embedded - Rewrite Sankey plugin with d3-sankey (replaces react-google-charts) - Upgrade Chord/Dependency plugins for Grafana 11.x/React 18 - Update ClickHouse operator to 0.27.0, ZooKeeper to 3.9.5 - Fix theia-manager for k8s apiserver compatibility (EffectiveVersion, OpenAPI, SingularNameProvider) - Add Antrea v2.6 columns to ClickHouse schema - Enable persistent storage for ClickHouse and ZooKeeper - Fully immutable install - no runtime downloads needed - Helm chart v0.9.0 Co-Authored-By: Oz <oz-agent@warp.dev>
Bump Go from 1.26.0 to 1.26.4 in go.mod and build/images/deps/go-version to pick up security fixes. The Makefile reads GO_VERSION from the deps file, so all Docker image builds will use 1.26.4 automatically. Co-Authored-By: Oz <oz-agent@warp.dev>
Add comprehensive README documentation including: - Requirements table (Kubernetes, Antrea, Go, Node.js, Docker, Helm) - Quick start deployment steps (5 steps) - Docker images table with all 8 images - Building instructions for Go images, Grafana image, upstream images - Configuration reference table with key Helm values - Architecture diagram showing data flow - Documentation links Co-Authored-By: Oz <oz-agent@warp.dev>
- Update pebble to latest in clickhouse-monitor and theia-manager Dockerfiles (apt-get install pebble to get latest from Ubuntu repos) - Fix CVE-2026-35469: upgrade github.com/moby/spdystream v0.5.0 -> v0.5.1 (DoS via SPDY streaming) - Fix CVE-2026-39883: upgrade go.opentelemetry.io/otel/sdk v1.40.0 -> v1.43.0 (PATH hijacking via BSD kenv) - Update vendor directory with fixed versions Co-Authored-By: Oz <oz-agent@warp.dev>
Root cause: migration file 000006_antrea-v2.6.up.sql used a non-numeric
version name. The schema management code splits filenames to extract
version strings and compares them numerically. 'antrea-v2.6' becomes
'antrea.v2.6' which fails strconv.Atoi("antrea"), crashing the
migration and preventing table creation.
Fixes:
- Rename 000006_antrea-v2.6.{up,down}.sql to 000006_0-9-0.{up,down}.sql
to follow the numeric <major>-<minor>-<patch> naming convention
- Add validation in initializeVersionMap() to skip files with
non-numeric version strings (defensive, prevents future breakage)
- Fix versionLessThan() to handle versions with different segment counts
- Tables (flows, materialized views, recommendations, tadetector) are
now created by the init.sh -> schema-management -> createTable flow
during ClickHouse pod startup, no manual intervention needed
Co-Authored-By: Oz <oz-agent@warp.dev>
Co-Authored-By: Oz <oz-agent@warp.dev>
Fix /clickhouse-schema-management path in init.sh (was ../clickhouse-schema-management which fails because init scripts run from /docker-entrypoint-initdb.d/ but the binary is at /clickhouse-schema-management in the image root). Verified: fresh ClickHouse deploy creates all tables automatically via init.sh -> /clickhouse-schema-management -> createTable flow. Co-Authored-By: Oz <oz-agent@warp.dev>
Create duplicate dashboards without kube-system/flow-visibility/flow-aggregator namespace exclusions, prefixed with 'all_ns_'. These show traffic from ALL namespaces including infrastructure namespaces. New dashboards: - all_ns_pod_to_pod_dashboard - all_ns_pod_to_service_dashboard - all_ns_pod_to_external_dashboard - all_ns_node_to_node_dashboard - all_ns_networkpolicy_dashboard - all_ns_network_topology_dashboard Each has a unique UID to avoid conflicts with the filtered originals. Original dashboards remain unchanged with their existing exclusions. Co-Authored-By: Oz <oz-agent@warp.dev>
Co-Authored-By: Oz <oz-agent@warp.dev>
The createTable function creates both ReplicatedMergeTree local tables
and Distributed tables. The distributed tables depend on ClickHouse
cluster macros ({cluster}, {shard}, {replica}) which may not be
available when the ClickHouse operator is still configuring the pod.
Add retry logic (5 attempts, 10s delay) to handle the race condition
where init.sh runs before the cluster macros are fully set up.
This fixes the 'Table default.flows does not exist' error that occurs
when the flow-aggregator starts before the distributed tables are
created.
Co-Authored-By: Oz <oz-agent@warp.dev>
Split the monolithic clickhouse-operator-install-bundle.yaml into: - crds/clickhouse-crds.yaml: 4 CRDs (applied once by Helm, not templated) - templates/clickhouse/operator.yaml: 17 operator resources (templated) The operator deployment now uses values from clickhouseOperator in values.yaml, allowing image overrides via --set or custom values files. Co-Authored-By: Oz <oz-agent@warp.dev>
Co-Authored-By: Oz <oz-agent@warp.dev>
Dashboard A - Denied Traffic Patterns: - Total denied connections and bytes (stat panels) - Denied traffic trend over time (bar chart) - Top 20 source namespaces by deny count - Top 20 destination IPs by deny count - Top 20 destination ports by deny count - Namespace-to-destination drilldown table (count + bytes) Dashboard B - Denied Policy Accountability: - Top denied egress policies by count and bytes - Top denied ingress policies by count and bytes - Denied rules over time (time series, per-policy) - Deny triage table with namespace, dest IP, port, direction, action (Drop/Reject), policy, count, bytes Both dashboards feature: - Direction dropdown (egress/ingress/both) defaulting to egress - Ad-hoc filters on flows_policy_view - 24h default time range, top 20 limits - Consistent deny filter: action IN (2=Drop, 3=Reject) - Destination IP only (no DNS, phase 1) Data source: flows_policy_view materialized view Time filter: flowEndSeconds via $__timeFilter Co-Authored-By: Oz <oz-agent@warp.dev>
…chart Both denied_traffic_patterns and denied_policy_accountability dashboards now have a Top N dropdown (default 20) that controls LIMIT on all ranked queries. Options: 20, 50, 100, 500. Co-Authored-By: Oz <oz-agent@warp.dev>
Dependency updates: - All direct Go deps updated to latest (clickhouse-go v2.47.0, k8s.io/* v0.36.3, grpc v1.83.0, crypto v0.54.0, etc.) - go mod tidy + go mod vendor re-run with updated deps - go.mod: Go 1.26.4 (matches installed toolchain) Minimal base images (zero/near-zero CVEs): - clickhouse-monitor: ubuntu:26.04 -> scratch (binary + CA certs only, ~17MB) - theia-manager: ubuntu:26.04 -> scratch (CGO_ENABLED=0, ~115MB) - clickhouse-server: 25.8 -> 25.8.28.1-distroless (latest patch, distroless) - spark-jobs: minimized apt footprint (wget removed post-install) SBOM + LICENSE embedded in every image: - /sbom.spdx.json: SPDX format generated by syft v1.50.0 during build - /LICENSE: Apache 2.0 copied into each final image Makefile: removed --sbom=true (SBOM embedded via syft instead) Co-Authored-By: Oz <oz-agent@warp.dev>
theia-grafana (built from source):
- Dockerfile.grafana: syft scans /opt/grafana, writes /sbom.spdx.json
and removes syft binary; copies LICENSE
Upstream/retag images (4 new wrapper Dockerfiles):
- Dockerfile.clickhouse-operator.wrapper
- Dockerfile.metrics-exporter.wrapper
- Dockerfile.clickhouse-server-upstream.wrapper
- Dockerfile.zookeeper.wrapper
Each wrapper uses ARG UPSTREAM_TAG so the Makefile can pass any
version tag, runs syft over the full filesystem, removes syft,
and copies LICENSE — giving every image /sbom.spdx.json + /LICENSE.
Makefile: pull-theia-{clickhouse-operator,metrics-exporter,
clickhouse-server-upstream,grafana,zookeeper} targets now call
'docker build' with the appropriate wrapper Dockerfile instead of
bare 'docker pull' + 'docker tag', so every pushed image contains
the SBOM and LICENSE files.
Co-Authored-By: Oz <oz-agent@warp.dev>
…n grafana
- Dockerfile.grafana: add ARG UPSTREAM_TAG=13.0.2 so pull-theia-grafana
target can inject any version tag at build time
- All 4 wrapper Dockerfiles: switch from single-stage (curl inside upstream,
fails on distroless) to multi-stage build:
stage 1 (upstream): copy upstream filesystem
stage 2 (alpine): install syft, run 'cd /upstream && syft .'
with corrected exclude patterns (./proc, ./sys, ./dev)
stage 3 (upstream): COPY sbom.spdx.json + LICENSE from alpine stage
Works for any upstream image including distroless variants.
Verified: /sbom.spdx.json and /LICENSE confirmed present in
theia-grafana, theia-clickhouse-server-upstream:25.8.28.1,
and theia-clickhouse-monitor (scratch).
Co-Authored-By: Oz <oz-agent@warp.dev>
Co-Authored-By: Oz <oz-agent@warp.dev>
The ClickHouse operator injects shell commands (/bin/sh -c ...) to run chown before starting the ClickHouse entrypoint. The distroless image variant does not include /bin/sh, causing containers to fail with: exec: "/bin/sh": stat /bin/sh: no such file or directory Revert clickhouse-server base from 25.8.28.1-distroless back to 25.8.28.1 (standard image with shell). Also bump Chart.yaml version/appVersion 0.9.0 -> 0.9.1 and grafana image tag v0.9.0 -> v0.9.1 in values.yaml. Verified: all pods Running in local cluster with v0.9.1 images. Co-Authored-By: Oz <oz-agent@warp.dev>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Comprehensive update to make Theia fully self-hosted with immutable deployments. All Docker images hosted on GHCR, Grafana plugins embedded in a custom image, and ClickHouse schema management fixed for automatic table creation.
Key Changes
Docker Images
ghcr.io/scoobed/withtheia-prefixtheia-grafanaimage (Grafana 13.0.2) with all plugins pre-installedGrafana Plugins
ClickHouse Schema Management
000006_antrea-v2.6→000006_0-9-0(numeric convention)../clickhouse-schema-management→/clickhouse-schema-managementTheia Manager
Helm Chart
Documentation
Testing
Warp conversation
Co-Authored-By: Oz oz-agent@warp.dev