What happened?
Problem
When deploying with replicaCount > 1, multiple operator pods run concurrently without leader election being effectively enabled, because the mounted controller_manager_config.yaml (which sets leaderElection.leaderElect: true) is never passed to the manager process.
Root cause
In fact , The Deployment starts /manager without --config=…, while the ConfigMap is mounted at /controller_manager_config.yaml. As a result, the binary falls back to CLI defaults (--leader-elect defaults to false), so no Lease objects are created and HA replicas act as concurrent controllers instead of leader / standby.
Did you expect to see something different?
having one leader instead of 2 concurent Pods
How to reproduce it (as minimally and precisely as possible):
Replica = 2 for operator and cass-operator
Environment
-
K8ssandra Operator version:
docker.io/k8ssandra/k8ssandra-operator@sha256:3fb182ff0317687682dacdfb272c5ad03440d12dd889965964951816b5de3acb
-
Kubernetes version information:
NAME STATUS ROLES AGE VERSION
oi-test-master-a1 Ready control-plane 324d v1.31.4
oi-test-master-b1 Ready control-plane 324d v1.31.4
oi-test-master-c1 Ready control-plane 324d v1.31.4
oi-test-worker-a1 Ready <none> 324d v1.31.4
oi-test-worker-a2 Ready <none> 324d v1.31.4
oi-test-worker-a3 Ready <none> 240d v1.31.4
oi-test-worker-a4 Ready <none> 175d v1.31.4
oi-test-worker-b1 Ready <none> 324d v1.31.4
oi-test-worker-b2 Ready <none> 324d v1.31.4
oi-test-worker-b3 Ready <none> 240d v1.31.4
oi-test-worker-b4 Ready <none> 175d v1.31.4
oi-test-worker-c1 Ready <none> 324d v1.31.4
oi-test-worker-c2 Ready <none> 324d v1.31.4
oi-test-worker-c3 Ready <none> 240d v1.31.4
oi-test-worker-c4 Ready <none> 175d v1.31.4
https://github.com/k8ssandra/k8ssandra-operator/blob/v1.31.0/charts/k8ssandra-operator/templates/deployment.yaml#L37
apiVersion: apps/v1
kind: Deployment
metadata:
name: {{ include "k8ssandra-common.fullname" . }}
labels: {{ include "k8ssandra-common.labels" . | indent 4 }}
control-plane: k8ssandra-operator
{{- with include "k8ssandra-common.annotations" . }}
annotations:
{{- . | nindent 4 }}
{{- end }}
spec:
replicas: {{ .Values.replicaCount }}
selector:
matchLabels:
{{- include "k8ssandra-common.selectorLabels" . | nindent 6 }}
control-plane: k8ssandra-operator
template:
metadata:
{{- with include "k8ssandra-common.annotations" (dict "context" . "annotations" .Values.podAnnotations) }}
annotations:
{{- . | nindent 8 }}
{{- end }}
labels:
{{- include "k8ssandra-common.labels" . | indent 8 }}
control-plane: k8ssandra-operator
spec:
{{- with .Values.imagePullSecrets }}
imagePullSecrets:
{{- toYaml . | nindent 8 }}
{{- end }}
serviceAccountName: {{ include "k8ssandra-common.serviceAccountName" . }}
securityContext:
{{- toYaml .Values.podSecurityContext | nindent 8 }}
containers:
- command:
- /manager
# args:
# - --config=controller_manager_config.yaml
BTW , i do not know why it is commented ? it seems to be the cause of the issue
insert K8ssandra Operator logs relevant to the issue here
Anything else we need to know?:
kubectl get lease -A | grep k8ssandra gives nothing and in the logs i see duplicate events .
so this confirm that there is no election
:Proposed change
Uncomment / add container args so the manager loads the mounted config file, e.g. --config=/controller_manager_config.yaml (path aligned with the volumeMount).
Optionally expose extraArgs / manager.extraArgs in values.yaml for flexibility without forking the chart.
What happened?
Problem
When deploying with replicaCount > 1, multiple operator pods run concurrently without leader election being effectively enabled, because the mounted controller_manager_config.yaml (which sets leaderElection.leaderElect: true) is never passed to the manager process.
Root cause
In fact , The Deployment starts /manager without --config=…, while the ConfigMap is mounted at /controller_manager_config.yaml. As a result, the binary falls back to CLI defaults (--leader-elect defaults to false), so no Lease objects are created and HA replicas act as concurrent controllers instead of leader / standby.
Did you expect to see something different?
having one leader instead of 2 concurent Pods
How to reproduce it (as minimally and precisely as possible):
Replica = 2 for operator and cass-operator
Environment
K8ssandra Operator version:
docker.io/k8ssandra/k8ssandra-operator@sha256:3fb182ff0317687682dacdfb272c5ad03440d12dd889965964951816b5de3acbKubernetes version information:
Kubernetes cluster kind:
insert how you created your cluster: Kubespray
Manifests:
https://github.com/k8ssandra/k8ssandra-operator/blob/v1.31.0/charts/k8ssandra-operator/templates/deployment.yaml#L37
BTW , i do not know why it is commented ? it seems to be the cause of the issue
Anything else we need to know?:
kubectl get lease -A | grep k8ssandragives nothing and in the logs i see duplicate events .so this confirm that there is no election
:Proposed change
Uncomment / add container args so the manager loads the mounted config file, e.g. --config=/controller_manager_config.yaml (path aligned with the volumeMount).
Optionally expose extraArgs / manager.extraArgs in values.yaml for flexibility without forking the chart.