Skip to content

Scheduler: AssigningResourceBindings cache leaks when Informer event arrives before Add() #7781

Description

@Priyanshu8023

What happened:
When the WorkloadAffinity feature gate is enabled, the scheduler adds a binding to the AssigningResourceBindings cache only after the API server patch succeeds. If the Informer delivers the update event for that patch (or the later FullyApplied event) before the Add() call runs, the cleanup handler finds no entry and drops the event. The entry added afterwards is then never cleaned up — it stays in memory until the binding is deleted.

The affected code is in patchScheduleResultForResourceBinding, where the race is already acknowledged in a TODO(@zhzhuang-zju) comment.

The cleanup path, OnBindingUpdate ,is a check-then-act pattern: it first checks whether an entry for the binding exists in the items map and returns immediately if it doesn't. This makes it a lost-notification race — an update event that arrives before Add() is silently discarded, and since Informer notifications are edge-triggered they are never redelivered. The terminal cleanup (deleting the entry once the binding is FullyApplied) is therefore unreachable for an entry inserted after its cleanup event was already consumed.

Key points:

  • GC() only sweeps the assumptions map — the items map hit by this race has no TTL and is never swept.
  • The GC goroutine only starts under SchedulingOvercommitProtection, so with only WorkloadAffinity enabled there is no fallback cleanup at all.
  • Leaked entries are fed into every scheduling cycle via GetBindings(), so completed bindings keep being scored as in-flight — a correctness cost, not just memory.
  • Growth is one entry per binding (removed on binding delete), so it accumulates steadily under churn in a long-running scheduler.

What you expected to happen:
Cleanup should work regardless of event/Add() ordering, and the items map should have a TTL safety net like the sibling assumptions map.

How to reproduce it (as minimally and precisely as possible):

  1. Deliver the Informer's FullyApplied update event to the cache before the scheduler's Add() call — the cache has no entry yet, so the event is dropped.
  2. Now call Add() — the entry is inserted.
  3. Observe: GC() removes nothing (it never sweeps this map) and GetBindings() still returns the entry. Nothing will ever remove it while the binding exists.
  4. Counter-check: run the same two calls in the opposite order (Add() first, then the event) — the entry is cleaned up correctly, proving the outcome depends purely on ordering.

Anything else we need to know?:

Environment:

  • Karmada version: master
  • kubectl-karmada or karmadactl version: N/A (code-level issue)
  • Others: requires the WorkloadAffinity feature gate

Metadata

Metadata

Assignees

Labels

kind/bugCategorizes issue or PR as related to a bug.

Type

No type

Projects

Status
No status

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions