Skip to content

fix(helm): wire controller manager config so leader election from ConfigMap is actually used #1723

Description

@boussoufiane

What happened?
Problem
When deploying with replicaCount > 1, multiple operator pods run concurrently without leader election being effectively enabled, because the mounted controller_manager_config.yaml (which sets leaderElection.leaderElect: true) is never passed to the manager process.

Root cause
In fact , The Deployment starts /manager without --config=…, while the ConfigMap is mounted at /controller_manager_config.yaml. As a result, the binary falls back to CLI defaults (--leader-elect defaults to false), so no Lease objects are created and HA replicas act as concurrent controllers instead of leader / standby.

Did you expect to see something different?
having one leader instead of 2 concurent Pods

How to reproduce it (as minimally and precisely as possible):
Replica = 2 for operator and cass-operator
Environment

  • K8ssandra Operator version:

    docker.io/k8ssandra/k8ssandra-operator@sha256:3fb182ff0317687682dacdfb272c5ad03440d12dd889965964951816b5de3acb

  • Kubernetes version information:

NAME                STATUS   ROLES           AGE    VERSION
oi-test-master-a1   Ready    control-plane   324d   v1.31.4
oi-test-master-b1   Ready    control-plane   324d   v1.31.4
oi-test-master-c1   Ready    control-plane   324d   v1.31.4
oi-test-worker-a1   Ready    <none>          324d   v1.31.4
oi-test-worker-a2   Ready    <none>          324d   v1.31.4
oi-test-worker-a3   Ready    <none>          240d   v1.31.4
oi-test-worker-a4   Ready    <none>          175d   v1.31.4
oi-test-worker-b1   Ready    <none>          324d   v1.31.4
oi-test-worker-b2   Ready    <none>          324d   v1.31.4
oi-test-worker-b3   Ready    <none>          240d   v1.31.4
oi-test-worker-b4   Ready    <none>          175d   v1.31.4
oi-test-worker-c1   Ready    <none>          324d   v1.31.4
oi-test-worker-c2   Ready    <none>          324d   v1.31.4
oi-test-worker-c3   Ready    <none>          240d   v1.31.4
oi-test-worker-c4   Ready    <none>          175d   v1.31.4
  • Kubernetes cluster kind:

    insert how you created your cluster: Kubespray

  • Manifests:

https://github.com/k8ssandra/k8ssandra-operator/blob/v1.31.0/charts/k8ssandra-operator/templates/deployment.yaml#L37

apiVersion: apps/v1
kind: Deployment
metadata:
  name: {{ include "k8ssandra-common.fullname" . }}
  labels: {{ include "k8ssandra-common.labels" . | indent 4 }}
    control-plane: k8ssandra-operator
  {{- with include "k8ssandra-common.annotations" . }}
  annotations:
    {{- . | nindent 4 }}
  {{- end }}
spec:
  replicas: {{ .Values.replicaCount }}
  selector:
    matchLabels:
      {{- include "k8ssandra-common.selectorLabels" . | nindent 6 }}
      control-plane: k8ssandra-operator
  template:
    metadata:
      {{- with include "k8ssandra-common.annotations" (dict "context" . "annotations" .Values.podAnnotations) }}
      annotations:
        {{- . | nindent 8 }}
      {{- end }}
      labels:
        {{- include "k8ssandra-common.labels" . | indent 8 }}
        control-plane: k8ssandra-operator
    spec:
      {{- with .Values.imagePullSecrets }}
      imagePullSecrets:
        {{- toYaml . | nindent 8 }}
      {{- end }}
      serviceAccountName: {{ include "k8ssandra-common.serviceAccountName" . }}
      securityContext:
        {{- toYaml .Values.podSecurityContext | nindent 8 }}
      containers:
      - command:
        - /manager
        # args:
        # - --config=controller_manager_config.yaml

BTW , i do not know why it is commented ? it seems to be the cause of the issue

  • K8ssandra Operator Logs:
insert K8ssandra Operator logs relevant to the issue here

Anything else we need to know?:

kubectl get lease -A | grep k8ssandra gives nothing and in the logs i see duplicate events .
so this confirm that there is no election

:Proposed change
Uncomment / add container args so the manager loads the mounted config file, e.g. --config=/controller_manager_config.yaml (path aligned with the volumeMount).
Optionally expose extraArgs / manager.extraArgs in values.yaml for flexibility without forking the chart.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions