fix(proxy): cache litellm model resolution to stop repeated Provider List spam

[connectsudhindra-gif]

Description

The proxy repeatedly prints LiteLLM's Provider List: https://docs.litellm.ai/docs/providers banner during normal operation, with no explanation or way to suppress it (#2851).

Root cause: _resolve_litellm_model() in headroom/proxy/savings_tracker.py runs on every savings-tracking update (i.e. every request). For any model LiteLLM can't price (a custom/local/gateway model name — e.g. the reporter's local oMLX setup), the uncached fallback path calls litellm.cost_per_token(...) purely to probe resolvability. When that probe fails, LiteLLM prints the banner as an internal side effect before raising, and since the probe was never cached, it re-fires on every single request for the same unresolvable model.

Closes #2851

Type of Change

• Bug fix (non-breaking change that fixes an issue)
• New feature (non-breaking change that adds functionality)
• Breaking change (fix or feature that would cause existing functionality to change)
• Documentation update
• Performance improvement
• Code refactoring (no functional changes)

Changes Made

• Added a module-level _model_resolution_cache: dict[str, str] in headroom/proxy/savings_tracker.py.
• Split _resolve_litellm_model() into a thin cache-checking wrapper plus _resolve_litellm_model_uncached() (the original logic, unchanged), so the LiteLLM probe runs at most once per distinct model name per process.
• Mirrors the existing _resolved_model_cache pattern already used for this same problem class in headroom/pricing/litellm_pricing.py.
• No behavior change for models LiteLLM can already price (fast path via model_cost lookup) — only the noisy uncached probe path is memoized.

Testing

• Unit tests pass (pytest)
• Linting passes (ruff check .)
• Type checking passes (mypy headroom) — not run; mypy isn't installed in this environment
• New tests added for new functionality — no new test added (see Real Behavior Proof below for why, and what I did verify instead)
• Manual testing performed

Test Output

$ python3 -m pytest tests/test_savings_tracker_zero_price.py -v
tests/test_savings_tracker_zero_price.py::test_compression_savings_zero_for_free_model PASSED
tests/test_savings_tracker_zero_price.py::test_compression_savings_falls_back_for_unknown_model PASSED
tests/test_savings_tracker_zero_price.py::test_compression_savings_uses_real_price_for_paid_model PASSED
tests/test_savings_tracker_zero_price.py::test_input_cost_zero_for_free_model PASSED
tests/test_savings_tracker_zero_price.py::test_output_savings_zero_for_free_model PASSED
tests/test_savings_tracker_zero_price.py::test_output_savings_falls_back_for_unknown_model PASSED
tests/test_savings_tracker_zero_price.py::test_output_savings_uses_real_price_for_paid_model PASSED
7 passed, 1 warning in 0.13s

$ python3 -m ruff check headroom/proxy/savings_tracker.py
All checks passed!

Real Behavior Proof

• Environment: macOS, Python 3.12.3, this repo checked out locally. Note: this sandbox does not have the compiled Rust extension (headroom._core) built (maturin develop was not run), so I could not start the actual proxy server or exercise this through a real HTTP request. headroom/proxy/savings_tracker.py itself has no dependency on the compiled extension, so it imports and runs standalone.
• Exact command / steps: wrote a standalone script that imports headroom.proxy.savings_tracker directly, installs a fake litellm module whose cost_per_token always raises (reproducing LiteLLM's real behavior of printing the banner then raising for an unresolvable model), and calls _estimate_compression_savings_usd("oMLX-custom-model", 100) 20 times in a loop to simulate 20 proxied requests against the same unpriced model.

  import types, importlib
  probe_calls = {"count": 0}
  def fake_cost_per_token(**kwargs):
      probe_calls["count"] += 1
      print("Provider List: https://docs.litellm.ai/docs/providers")
      raise ValueError("LLM Provider NOT provided")
  fake_litellm = types.SimpleNamespace(model_cost={}, cost_per_token=fake_cost_per_token)
  st = importlib.import_module("headroom.proxy.savings_tracker")
  st.LITELLM_AVAILABLE = True
  st.litellm = fake_litellm
  for _ in range(20):
      st._estimate_compression_savings_usd("oMLX-custom-model", 100)
  print(f"probe fired {probe_calls['count']} time(s)")

• Observed result: on main (before this fix, verified via git stash/checkout) the banner printed 20 times for 20 simulated requests — once per request. On this branch (after the fix), the banner printed 1 time total across all 20 simulated requests — confirming the probe now runs once per model per process instead of once per request.
• Not tested: the real end-to-end path (actual headroom proxy process handling live HTTP requests against a genuinely unpriced model) — blocked by the missing compiled headroom._core extension in this sandbox, as noted above. The savings-tracking call path exercised here (_estimate_compression_savings_usd → _resolve_litellm_model) is the same code path the real proxy calls per-request from headroom/proxy/server.py, so I'm confident this generalizes, but I have not confirmed it against the live proxy myself.

Review Readiness

• I have performed a self-review
• This PR is ready for human review

Checklist

• My code follows the project's style guidelines
• I have performed a self-review of my code
• I have commented my code, particularly in hard-to-understand areas — added a comment explaining the cache's purpose; no other hard-to-understand areas introduced
• I have made corresponding changes to the documentation — N/A, internal implementation detail with no user-facing API/doc surface
• My changes generate no new warnings
• I have added tests that prove my fix is effective or that my feature works — not added as an automated test (see Real Behavior Proof; the existing tests/test_savings_tracker_zero_price.py suite already covers _resolve_litellm_model's callers and continues to pass unchanged)
• New and existing unit tests pass locally with my changes
• I did not edit CHANGELOG.md

Additional Notes

• Full repo test suite was not run locally: several test modules (test_proxy_savings_history.py, test_gateway_sidecar_ports.py, etc.) require the compiled headroom._core Rust extension, which isn't built in this sandbox (ModuleNotFoundError: No module named 'headroom._core'). Confirmed via git stash that this is a pre-existing environment gap unrelated to this change.
• I manually traced tests/test_proxy_savings_history.py::test_litellm_resolution_and_savings_estimation_fallbacks (which exercises _resolve_litellm_model with a mutated model_cost dict across several assertions in one test) against the new caching behavior and confirmed no asserted value changes, since the cache only memoizes the resolved model name string, not the price lookups (which stay live).
• Happy to add an automated regression test (e.g. asserting litellm.cost_per_token call count) if a maintainer points me to where they'd want it.

[github-actions]

PR governance

This PR follows the template and is marked ready for human review.

[JerrettDavis]

Caching the failed probe addresses the repeated banner, but the cache needs a bound before it is safe on a request-facing proxy. The model name is client-controlled, and _model_resolution_cache is a process-lifetime plain dict. A caller can send a new arbitrary model string on every request and grow it without limit; this turns a log-noise fix into an easy memory-retention path. The similar existing cache is not a reason to duplicate that exposure.

Please use a bounded cache (with a deliberate size/eviction policy) and add the automated regression test already outlined in the PR: repeated resolution of one unknown model probes once, while enough distinct names demonstrate the cache remains bounded and an evicted name can be resolved again. That test should also clear/isolate cache state so unrelated pricing tests remain order-independent.