What happened?
On OpenShift, the k8ssandra-operator cannot create the Cassandra Prometheus ServiceMonitor. The telemetry reconcile fails with error:
servicemonitors.monitoring.coreos.com "<cluster>-<dc>-cass-servicemonitor" is forbidden: cannot set blockOwnerDeletion if an ownerReference refers to a resource you can't set finalizers on: , <nil>
Did you expect to see something different?
Yes — the ServiceMonitor should be created and Cassandra metrics scraped, as happens on non-OpenShift clusters (where the admission plugin is not enabled). The K8ssandraCluster is otherwise healthy; only the telemetry step fails.
How to reproduce it (as minimally and precisely as possible):
- Use a cluster with the
OwnerReferencesPermissionEnforcement admission plugin enabled (default on OpenShift/OKD).
- Install the Prometheus Operator so the
ServiceMonitor CRD is present.
- Install k8ssandra-operator via Helm.
- Create a
K8ssandraCluster with Cassandra Prometheus telemetry enabled (see manifest below).
- Observe the operator repeatedly fail to create the cass
ServiceMonitor with the forbidden ... blockOwnerDeletion error; no ServiceMonitor is created.
Environment
-
K8ssandra Operator version:
v1.32.3
-
Kubernetes version information:
oc version Client Version: 4.19.4 Kustomize Version: v5.5.0 Kubernetes Version: v1.31.14
-
Kubernetes cluster kind:
OpenShift (OKD/OCP, ~4.18), namespace-scoped operator install via Helm
-
Manifests:
apiVersion: k8ssandra.io/v1alpha1
kind: K8ssandraCluster
metadata:
name: demo
spec:
cassandra:
serverVersion: "4.1.11"
telemetry:
prometheus:
enabled: true
datacenters:
- metadata:
name: dc1
size: 3
ERROR could not create ServiceMonitor resource ... "error":
"servicemonitors.monitoring.coreos.com \"demo-dc1-cass-servicemonitor\" is forbidden:
cannot set blockOwnerDeletion if an ownerReference refers to a resource you can't set
finalizers on: , <nil>"
pkg/telemetry.PrometheusResourcer.UpdateResources
/workspace/pkg/telemetry/prometheus_resourcer.go:64
controllers/k8ssandra.(*K8ssandraClusterReconciler).reconcileCassandraDCTelemetry
/workspace/controllers/k8ssandra/cassandra_telemetry_reconciler.go:73
controllers/k8ssandra.(*K8ssandraClusterReconciler).Reconcile
/workspace/controllers/k8ssandra/k8ssandracluster_controller.go:96
Anything else we need to know?:
This is the same class of issue as #1609, which was resolved in #1611 by switching from SetControllerReference to SetOwnerReference in the Medusa controllers. The telemetry ServiceMonitor path follows the same ownership pattern (an object owned by the CassandraDatacenter with blockOwnerDeletion), so it can probably be resolved in the same way. In the meantime a quick workaround is to patch the role and add the finalizers permission of the cassandradatacenters resource.
What happened?
On OpenShift, the k8ssandra-operator cannot create the Cassandra Prometheus ServiceMonitor. The telemetry reconcile fails with error:
servicemonitors.monitoring.coreos.com "<cluster>-<dc>-cass-servicemonitor" is forbidden: cannot set blockOwnerDeletion if an ownerReference refers to a resource you can't set finalizers on: , <nil>Did you expect to see something different?
Yes — the
ServiceMonitorshould be created and Cassandra metrics scraped, as happens on non-OpenShift clusters (where the admission plugin is not enabled). TheK8ssandraClusteris otherwise healthy; only the telemetry step fails.How to reproduce it (as minimally and precisely as possible):
OwnerReferencesPermissionEnforcementadmission plugin enabled (default on OpenShift/OKD).ServiceMonitorCRD is present.K8ssandraClusterwith Cassandra Prometheus telemetry enabled (see manifest below).ServiceMonitorwith theforbidden ... blockOwnerDeletionerror; noServiceMonitoris created.Environment
K8ssandra Operator version:
v1.32.3Kubernetes version information:
oc version Client Version: 4.19.4 Kustomize Version: v5.5.0 Kubernetes Version: v1.31.14Kubernetes cluster kind:
OpenShift (OKD/OCP, ~4.18), namespace-scoped operator install via Helm
Manifests:
Anything else we need to know?:
This is the same class of issue as #1609, which was resolved in #1611 by switching from
SetControllerReferencetoSetOwnerReferencein the Medusa controllers. The telemetryServiceMonitorpath follows the same ownership pattern (an object owned by theCassandraDatacenterwithblockOwnerDeletion), so it can probably be resolved in the same way. In the meantime a quick workaround is to patch the role and add the finalizers permission of thecassandradatacentersresource.