Skip to content

Cassandra ServiceMonitor creation fails on OpenShift (blockOwnerDeletion) #1757

Description

@mscerri

What happened?

On OpenShift, the k8ssandra-operator cannot create the Cassandra Prometheus ServiceMonitor. The telemetry reconcile fails with error:

servicemonitors.monitoring.coreos.com "<cluster>-<dc>-cass-servicemonitor" is forbidden: cannot set blockOwnerDeletion if an ownerReference refers to a resource you can't set finalizers on: , <nil>

Did you expect to see something different?

Yes — the ServiceMonitor should be created and Cassandra metrics scraped, as happens on non-OpenShift clusters (where the admission plugin is not enabled). The K8ssandraCluster is otherwise healthy; only the telemetry step fails.

How to reproduce it (as minimally and precisely as possible):

  1. Use a cluster with the OwnerReferencesPermissionEnforcement admission plugin enabled (default on OpenShift/OKD).
  2. Install the Prometheus Operator so the ServiceMonitor CRD is present.
  3. Install k8ssandra-operator via Helm.
  4. Create a K8ssandraCluster with Cassandra Prometheus telemetry enabled (see manifest below).
  5. Observe the operator repeatedly fail to create the cass ServiceMonitor with the forbidden ... blockOwnerDeletion error; no ServiceMonitor is created.

Environment

  • K8ssandra Operator version:

    v1.32.3

  • Kubernetes version information:

    oc version Client Version: 4.19.4 Kustomize Version: v5.5.0 Kubernetes Version: v1.31.14

  • Kubernetes cluster kind:

    OpenShift (OKD/OCP, ~4.18), namespace-scoped operator install via Helm

  • Manifests:

apiVersion: k8ssandra.io/v1alpha1
kind: K8ssandraCluster
metadata:
  name: demo
spec:
  cassandra:
    serverVersion: "4.1.11"
    telemetry:
      prometheus:
        enabled: true
    datacenters:
      - metadata:
          name: dc1
        size: 3
  • K8ssandra Operator Logs:
ERROR could not create ServiceMonitor resource ... "error":

"servicemonitors.monitoring.coreos.com \"demo-dc1-cass-servicemonitor\" is forbidden:
cannot set blockOwnerDeletion if an ownerReference refers to a resource you can't set
finalizers on: , <nil>"
pkg/telemetry.PrometheusResourcer.UpdateResources
    /workspace/pkg/telemetry/prometheus_resourcer.go:64
controllers/k8ssandra.(*K8ssandraClusterReconciler).reconcileCassandraDCTelemetry
    /workspace/controllers/k8ssandra/cassandra_telemetry_reconciler.go:73
controllers/k8ssandra.(*K8ssandraClusterReconciler).Reconcile
    /workspace/controllers/k8ssandra/k8ssandracluster_controller.go:96

Anything else we need to know?:

This is the same class of issue as #1609, which was resolved in #1611 by switching from SetControllerReference to SetOwnerReference in the Medusa controllers. The telemetry ServiceMonitor path follows the same ownership pattern (an object owned by the CassandraDatacenter with blockOwnerDeletion), so it can probably be resolved in the same way. In the meantime a quick workaround is to patch the role and add the finalizers permission of the cassandradatacenters resource.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions