What happened:
When the WorkloadAffinity feature gate is enabled, the scheduler adds a binding to the AssigningResourceBindings cache only after the API server patch succeeds. If the Informer delivers the update event for that patch (or the later FullyApplied event) before the Add() call runs, the cleanup handler finds no entry and drops the event. The entry added afterwards is then never cleaned up — it stays in memory until the binding is deleted.
The affected code is in patchScheduleResultForResourceBinding, where the race is already acknowledged in a TODO(@zhzhuang-zju) comment.
The cleanup path, OnBindingUpdate ,is a check-then-act pattern: it first checks whether an entry for the binding exists in the items map and returns immediately if it doesn't. This makes it a lost-notification race — an update event that arrives before Add() is silently discarded, and since Informer notifications are edge-triggered they are never redelivered. The terminal cleanup (deleting the entry once the binding is FullyApplied) is therefore unreachable for an entry inserted after its cleanup event was already consumed.
Key points:
GC() only sweeps the assumptions map — the items map hit by this race has no TTL and is never swept.
- The GC goroutine only starts under
SchedulingOvercommitProtection, so with only WorkloadAffinity enabled there is no fallback cleanup at all.
- Leaked entries are fed into every scheduling cycle via
GetBindings(), so completed bindings keep being scored as in-flight — a correctness cost, not just memory.
- Growth is one entry per binding (removed on binding delete), so it accumulates steadily under churn in a long-running scheduler.
What you expected to happen:
Cleanup should work regardless of event/Add() ordering, and the items map should have a TTL safety net like the sibling assumptions map.
How to reproduce it (as minimally and precisely as possible):
- Deliver the Informer's
FullyApplied update event to the cache before the scheduler's Add() call — the cache has no entry yet, so the event is dropped.
- Now call
Add() — the entry is inserted.
- Observe:
GC() removes nothing (it never sweeps this map) and GetBindings() still returns the entry. Nothing will ever remove it while the binding exists.
- Counter-check: run the same two calls in the opposite order (
Add() first, then the event) — the entry is cleaned up correctly, proving the outcome depends purely on ordering.
Anything else we need to know?:
Environment:
- Karmada version: master
- kubectl-karmada or karmadactl version: N/A (code-level issue)
- Others: requires the
WorkloadAffinity feature gate
What happened:
When the
WorkloadAffinityfeature gate is enabled, the scheduler adds a binding to theAssigningResourceBindingscache only after the API server patch succeeds. If the Informer delivers the update event for that patch (or the laterFullyAppliedevent) before theAdd()call runs, the cleanup handler finds no entry and drops the event. The entry added afterwards is then never cleaned up — it stays in memory until the binding is deleted.The affected code is in
patchScheduleResultForResourceBinding, where the race is already acknowledged in aTODO(@zhzhuang-zju)comment.The cleanup path,
OnBindingUpdate,is a check-then-act pattern: it first checks whether an entry for the binding exists in theitemsmap and returns immediately if it doesn't. This makes it a lost-notification race — an update event that arrives beforeAdd()is silently discarded, and since Informer notifications are edge-triggered they are never redelivered. The terminal cleanup (deleting the entry once the binding isFullyApplied) is therefore unreachable for an entry inserted after its cleanup event was already consumed.Key points:
GC()only sweeps theassumptionsmap — theitemsmap hit by this race has no TTL and is never swept.SchedulingOvercommitProtection, so with onlyWorkloadAffinityenabled there is no fallback cleanup at all.GetBindings(), so completed bindings keep being scored as in-flight — a correctness cost, not just memory.What you expected to happen:
Cleanup should work regardless of event/
Add()ordering, and theitemsmap should have a TTL safety net like the siblingassumptionsmap.How to reproduce it (as minimally and precisely as possible):
FullyAppliedupdate event to the cache before the scheduler'sAdd()call — the cache has no entry yet, so the event is dropped.Add()— the entry is inserted.GC()removes nothing (it never sweeps this map) andGetBindings()still returns the entry. Nothing will ever remove it while the binding exists.Add()first, then the event) — the entry is cleaned up correctly, proving the outcome depends purely on ordering.Anything else we need to know?:
Environment:
WorkloadAffinityfeature gate