Skip to content

feat(router): same-model cross-provider fallback - #932

Open
stephenzwj wants to merge 1 commit into
tashfeenahmed:mainfrom
stephenzwj:same-model-cross-provider-fallback
Open

feat(router): same-model cross-provider fallback#932
stephenzwj wants to merge 1 commit into
tashfeenahmed:mainfrom
stephenzwj:same-model-cross-provider-fallback

Conversation

@stephenzwj

Copy link
Copy Markdown

What it does

When a client pins an explicit model and that same model_id is served by multiple enabled platforms, failed attempts now try sibling rows for the same model immediately after the pinned row, before falling through to unrelated models.

Previously, a failed explicit pin could fall through to whichever unrelated model scored next in the chain.

Scope

  • Request-scoped: only applies when preferredModelDbId is set.
  • Auto and sticky routing behavior is unchanged.
  • No-op when the pinned model has no same-model siblings.

Tests

Adds four regression tests covering:

  • pinned model remains first;
  • same-model sibling is selected before an unrelated model;
  • unrelated model is selected after all same-model rows are exhausted;
  • no-sibling behavior remains unchanged.

Full server test suite: 214 files, 2,352 tests passed, 5 skipped.
Server build passes.

When a client pins an explicit model and that model_id is also served by other enabled platforms, try those sibling rows immediately after the pinned one on fallback, ahead of the rest of the chain. Request-scoped so auto and sticky routing remain unchanged.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant