fix: exclude capacity.json from Cruise Control pod-template hash by default

[gomitrah]

Description

Fixes #301

Two independent controllers race during broker scaling:

1. capacity.json is regenerated on every broker add/remove and, by default, its content is
   hashed into the Cruise Control Deployment's pod-template annotations
   (GeneratePodAnnotations in pkg/resources/cruisecontrol/deployment.go). Any hash change
   causes Kubernetes to roll the Cruise Control pod.
2. CruiseControlOperationReconciler submits add_broker/remove_broker operations based
   solely on Cruise Control's own REST API readiness (CruiseControlStatus.IsReady()), with no
   awareness of the Deployment's rollout state.

When these overlap, the fresh Cruise Control pod loses the in-memory task for an in-flight scale
operation. The operation is marked errored and retried after a fixed delay plus however long CC
takes to re-warm its metrics window — stalling scaling for anywhere from seconds to minutes.

Verified this is reachable on current master, independent of #300 (a KRaft-hardening PR whose
only CC-adjacent change is unrelated node filtering in remove_broker) — confirmed via a Go test
driving the real GenerateCapacityConfig + GeneratePodAnnotations functions for a 3→4 broker
scale-up, showing the pod-template hash changes purely from the broker-count change.

capacity.json is only read by Cruise Control at process startup, and the operator already
falls back to CC's own capacity estimation for scale operations, so a capacity.json change does
not require restarting a running CC pod. GeneratePodAnnotations now excludes capacity.json
from the pod-template hash by default, inverting the polarity of a pre-existing (undocumented,
never-set-by-default) escape hatch:

// before: hashed by default, excluded only when explicitly opted out
if value, ok := annotations[capacityConfigAnnotation]; !ok || value == "static" { ... }

// after: excluded by default, hashed only when explicitly opted in
if value, ok := annotations[capacityConfigAnnotation]; ok && value == "static" { ... }

Operators who need exact (non-estimated) capacity for newly added brokers immediately, and are
willing to accept the restart-race trade-off, can opt back in by setting
cruise-control.banzaicloud.com/broker-capacity-config: "static".

Upgrade note: every existing cluster has this annotation unset today, so the old code always
hashed capacity.json. The first reconcile after upgrading will drop the
cruiseControlCapacity.json pod-template annotation, which Kubernetes will apply as one
Deployment update — i.e. every existing cluster gets exactly one Cruise Control pod restart on
upgrade. This is expected and one-time, not a bug (see the upgrade-transition unit test).

Scope: this closes the specific capacity.json trigger described in #301. The other three
config sources hashed here (cruisecontrol.properties, clusterConfigs.json,
log4j.properties) still unconditionally roll the Deployment on change and can still race an
in-flight operation via the same mechanism — that's a separate, larger fix (giving the operation
controller Deployment-rollout awareness) and is intentionally out of scope here.

Type of Change

• Bug Fix
• New Feature
• Breaking Change
• Refactor
• Documentation
• Other (please describe)

Checklist

• I have read the contributing guidelines
• Existing issues have been referenced (where applicable)
• I have verified this change is not present in other open pull requests
• Functionality is documented
• All code style checks pass
• New code contribution is covered by automated tests
• All new and existing tests pass

Testing

• Unit tests (pkg/resources/cruisecontrol/deployment_test.go): default/unset, nil map, exact
  "static" opt-in, near-miss values (empty string, wrong case, unrelated value), passthrough of
  unrelated annotations, the exact scale-up scenario from CruiseControl restart races in-flight add_broker/remove_broker operations (scaling stalls during CC roll) #301, the static opt-in still
  restarting on change, and the one-time upgrade transition.
• Integration-style test (pkg/resources/cruisecontrol/cruisecontrol_reconcile_test.go): drives
  the real Reconciler.Reconcile against a fake Kubernetes API to confirm the static opt-in
  survives a full reconcile pass.
• Envtest (controllers/tests/kafkacluster_controller_test.go): new scenario exercising the
  static opt-in through the real controller + real API server, alongside updated assertions in
  kafkacluster_controller_cruisecontrol_test.go for the new default.
• go build ./..., go vet ./..., gofmt, and golangci-lint all clean (pre-existing
  goconst findings in configmap.go unrelated to this diff, confirmed present on master too).
• Full existing suite (pkg/resources/cruisecontrol, controllers, controllers/tests,
  controllers/tests/clusterregistry, controllers/tests/contourwatch, pkg/scale) run before
  and after the change — pass/fail status identical except for the intentional assertion updates
  above; no other workflow regressed.

Before fix: scale-up → capacity.json changes → CC pod restarts → in-flight operation lost →
stalls for 30s+ retry delay plus CC re-warm time.
After fix: scale-up → capacity.json changes → CC pod template unchanged → no restart →
operation proceeds uninterrupted.

[amuraru]

The problem analysis here is correct — the CC-roll ↔ in-flight-operation race in #301 is real, and the two-controller root cause is accurately described. But the chosen fix (stop hashing capacity.json into the CC pod template, so a capacity change no longer rolls CC) fixes broker removal while breaking broker add (upscale). Requesting changes.

Why the flip breaks upscale

The CC roll that this PR removes is also what loads a newly-added broker's capacity into a running CruiseControl. CC does not pick it up any other way:

• BrokerCapacityConfigFileResolver reads capacity.json once at startup (configure()); it does not watch/reload the file.
• When a broker is missing from the loaded map, capacityForBroker does not estimate from peers — it returns the broker-id -1 default entry, and if there is no -1 entry it NPEs (capacitiesForBrokers.get(-1).capacity() on null); with allowCapacityEstimation=false it throws BrokerCapacityResolutionException:

// BrokerCapacityConfigFileResolver.capacityForBroker (~L169-190)
BrokerCapacityInfo capacity = capacitiesForBrokers.get(brokerId);
if (capacity != null) return capacity;
if (allowCapacityEstimation) {
  return new BrokerCapacityInfo(capacitiesForBrokers.get(DEFAULT_CAPACITY_BROKER_ID).capacity(), ...); // -1 default; NPE if absent
} else {
  throw new BrokerCapacityResolutionException(...);
}

• koperator's GenerateCapacityConfig (pkg/resources/cruisecontrol/configmap.go) emits a -1 default only if the user supplies spec.cruiseControlConfig.capacityConfig with one; by default it generates per-broker entries with no -1. None of the shipped samples set a -1 except the kitchen-sink config/samples/banzaicloud_v1beta1_kafkacluster.yaml (and that one is a non-JBOD scalar DISK, not the per-log-dir map JBOD/KRaft brokers use). The KRaft samples and every e2e sample have no -1.

Consequence: with this change, adding a broker no longer rolls CC, so the running CC (started before the new broker) never learns the new broker's capacity — not in its startup capacity.json, no live reload, no -1 default → capacityForBroker(newBroker) NPEs / throws, CC can't build a model that includes the broker, and the upscale stalls with no roll to recover. That's a regression versus master, where the capacity-driven roll makes a fresh CC read the new broker's exact per-broker entry. Removal is unaffected (the departing broker leaves cluster metadata, and the remaining brokers are already in CC's startup capacity.json) — which is why a remove-only test passes and hides this.

Suggested direction

Keep the capacity-driven roll (so CC always gets exact per-broker capacity, with no dependency on estimation or a -1 default), and instead sequence it against operations:

• gate the CruiseControlOperationReconciler so it does not submit/track an add_broker/remove_broker while the CC Deployment is mid-rollout (not settled to a single Ready replica — the isCruiseControlDeploymentRolledOut check already used in the e2e), and
• avoid rolling CC while a scaling operation is in flight,

so the ordering becomes: capacity change → roll CC → CC settles/re-warms → submit op → op runs against a stable CC. This preserves exact capacity for homogeneous and heterogeneous clusters and removes the estimation/-1 dependency entirely.

For transparency: the other #301 PR (#304) took this same flip approach and is being reworked to the sequencing approach above for the same reason. Whichever lands should include an add_broker e2e, since a remove-only scenario does not exercise the broken path.

[amuraru]

dupe of #304