Replica churn in member clusters when workload replicas and PropagationPolicy weights are updated together

[ryanwuer]

What happened:

A workload's replica count and its placement live on two different API objects: the replica count on the workload itself, and the weights on the PropagationPolicy. When both need to change together, a GitOps tool applies them as two separate API calls with no ordering guarantee.

During the window between those two calls, the detector can observe a new value on one object and a stale value on the other. ApplyPolicy then writes that pair into the ResourceBinding and the scheduler immediately acts on it, producing real replica churn in the member clusters.

Concrete example. Two member clusters, staticWeightList A=1, B=1, workload replicas: 4 → 2 replicas in each cluster. The goal is to drain cluster A to 0 while cluster B keeps exactly 2, which requires two coordinated edits:

• PropagationPolicy: weights A=1, B=1 → A=1, B=10000
• workload: replicas: 4 → 2

Depending on which call lands first, the scheduler observes one of:

Order	Transient state seen by scheduler	Result in cluster B
PropagationPolicy first	replicas=4 + new weights	2 → 4 (doubles)
Workload first	replicas=2 + old weights 1:1	2 → 1 (halves)

Both are transient, but they are not harmless: the first doubles the resource footprint of the surviving cluster, and the second halves capacity at exactly the moment that cluster is about to take over all traffic. At scale (hundreds of workloads drained in one batch) hitting the second case on even a small fraction is disruptive.

What you expected to happen:

The scheduler settles on the final state instead of acting on every intermediate combination. A short burst of ResourceBinding spec updates should collapse into a single scheduling pass.

How to reproduce it (as minimally and precisely as possible):

1. Propagate a workload to two member clusters with replicaSchedulingType: Divided, replicaDivisionPreference: Weighted, and staticWeightList A=1, B=1. Set replicas: 4 and wait for 2 / 2.

2. In a single commit, change the PropagationPolicy weights to A=1, B=10000 and the workload to replicas: 2.

3. Let a GitOps tool sync both objects, then watch the allocation:

   kubectl get resourcebinding -n <ns> <name> -o jsonpath='{.spec.clusters}' -w
   

The final state is always A=0, B=2. Whether an intermediate B=4 or B=1 is materialized depends on whether the scheduler runs between the two applies — i.e. on timing, not on anything the user controls.

Anything else we need to know?:

The ResourceBinding write itself is already atomic — that is not the gap. ApplyPolicy builds Replicas and Placement in one shot and writes them in a single CreateOrUpdate:

// pkg/detector/detector.go — BuildResourceBinding
Placement: &policySpec.Placement,                               // from PropagationPolicy
replicas, _, err := d.ResourceInterpreter.GetReplicas(object)   // from the workload
propagationBinding.Spec.Replicas = replicas

// pkg/detector/detector.go — ApplyPolicy
bindingCopy.Spec.Replicas  = binding.Spec.Replicas
bindingCopy.Spec.Placement = binding.Spec.Placement

Atomic writing does not imply a consistent pair: both fields come from the same reconcile, but that reconcile may read a fresh version of one source and a stale version of the other. The resulting inconsistent pair is then written atomically and scheduled.

Why the ordering cannot be relied on, using Argo CD as the example (gitops-engine pkg/sync/sync_tasks.go):

// we take advantage of the fact that if the kind is not in the kindOrder map,
// then it will return the default int value of zero, which is the highest value
d = kindOrder[a.GetKind()] - kindOrder[b.GetKind()]
if d != 0 { return d < 0 }
return a.GetName() < b.GetName()

kindOrder only covers built-in kinds, so PropagationPolicy and most workload CRDs both fall back to 0. If the two objects also share a name — common when a chart derives both from the same release name — the comparison is a tie, and sort.Sort is not stable, so the relative order is effectively arbitrary. runTasks then groups by kind and runs different kinds serially, so the two applies are separated by a full API round trip rather than being back-to-back.

Note this is not Argo CD specific: any tool that applies a workload and its PropagationPolicy as separate requests has the same exposure. Ordering hints (sync waves and similar) only pick which of the two transient states occurs — they cannot remove the window.

Proposal

Add an optional debounce on ResourceBinding spec updates so a burst of updates collapses into one scheduling pass. Three properties of the existing code make this a small change:

1. AddAfter keeps the earliest ready time for a key (insert() in client-go's delaying queue), so later updates within the window are absorbed rather than extending it.
2. doScheduleBinding re-reads the binding from the lister, so when the delayed key is finally processed it observes the settled state — not the object captured in the event.
3. PriorityBasedScheduling is Alpha and off by default, so only the legacy queue path needs to change.

// pkg/scheduler/event_handler.go — onResourceBindingUpdate
if s.bindingUpdateDebounce > 0 {
    s.queue.AddAfter(key, s.bindingUpdateDebounce)
} else {
    s.queue.Add(key)   // default 0 preserves current behaviour
}

Only update events would be delayed; adds and requeues (failover, cluster changes) stay on the fast path. Exposed as a --binding-update-debounce flag defaulting to 0, so behaviour is unchanged unless explicitly enabled.

I have a working patch against v1.14.5 (4 files, ~58 added lines) that builds and passes go vet, with unit tests covering both the collapsed and non-collapsed cases. Happy to open a PR if the maintainers agree with the direction — and equally happy to be pointed at an existing mechanism if I have missed one.

Environment:

• Karmada version: v1.14.5
• kubectl-karmada or karmadactl version: n/a (reproduced against a v1.14.5 control plane)
• Others: observed with Argo CD as the GitOps tool; the analysis above suggests it is not tool-specific.

[RainbowMango]

Add an optional debounce on ResourceBinding spec updates so a burst of updates collapses into one scheduling pass.

I don't quite understand how it works. It looks like you are proposing an update window, which means the update event on ResourceBinding will not be handled immediately, but will be held for a while. In my opinion, it still cannot guarantee that two sequential update events could be in the same window.

I think the Suspension feature might fit your case; please look at https://karmada.io/docs/userguide/scheduling/propagation-policy/#suspension for more details. You can suspend the propagation, do your ops, and then resume propagation after all changes reflect in ResourceBinding.

[ryanwuer]

I don't quite understand how it works. It looks like you are proposing an update window, which means the update event on ResourceBinding will not be handled immediately, but will be held for a while. In my opinion, it still cannot guarantee that two sequential update events could be in the same window.

The yaml contains replicas and the yaml contains weight are deployed step by step in the same sync wave of ArgoCD, the time gap duration is quite small, maybe 10ms. From our testing result, the problem is gone when update window is set to 2s.

The problem also relates to Suspension. Assuming we have 10 replicas, a: 5 and b: 5, now we want to scale b to 0, the following steps need to be done at the same time:

1. suspend a
2. change 10 replicas to 0 replicas

Suspend and replicas exist in different yaml, deployed step by step by ArgoCD in the same sync wave, the time gap is small, like 10ms. But karmada-scheduler could see the replicas is set to 0 but not see the suspend change at a specific moment, as a result, a and b are scaled to 0.