Skip to content

feat: add LiteLLM as AI gateway provider#18

Open
RheagalFire wants to merge 1 commit into
lynote-ai:mainfrom
RheagalFire:feat/add-litellm-provider
Open

feat: add LiteLLM as AI gateway provider#18
RheagalFire wants to merge 1 commit into
lynote-ai:mainfrom
RheagalFire:feat/add-litellm-provider

Conversation

@RheagalFire

@RheagalFire RheagalFire commented Jun 11, 2026

Copy link
Copy Markdown

Summary

Adds LiteLLM as a third provider alongside DeepSeek and OpenRouter, giving users access to 100+ LLM providers (Anthropic, OpenAI, Bedrock, Vertex AI, Groq, etc.) through a single provider = "litellm" config change.

Motivation

The current provider system supports DeepSeek and OpenRouter. LiteLLM adds 100+ providers via its SDK without needing a separate proxy. Users set provider = "litellm" in config.toml, pick any model string (e.g. anthropic/claude-sonnet-4-6), and LiteLLM routes to the right provider using standard env vars (ANTHROPIC_API_KEY, OPENAI_API_KEY, etc.). drop_params=True handles cross-provider kwarg incompatibilities automatically.

Changes

  • src/standard/llm_client.py -- added "litellm" to PROVIDER_DEFAULTS, new _litellm_chat_completions() function using litellm.completion() with drop_params=True, provider kwarg threaded through chat_completions()
  • src/standard/llm_rewriter.py -- forwards provider kwarg
  • src/standard/pipeline.py -- forwards provider kwarg to both rewrite steps
  • src/methodologies/llm_rewriter.py -- forwards provider kwarg

Tests

Existing tests (12/12 pass):

tests/test_llm_client.py  12 passed in 0.02s

Live E2E (LiteLLM SDK -> Azure Foundry -> Claude Sonnet 4.6):

Response: '4'
=== E2E PASSED ===

Example usage

# config.toml
[llm]
provider = "litellm"
model = "anthropic/claude-sonnet-4-6"
# LiteLLM reads provider API keys from env vars
export ANTHROPIC_API_KEY="sk-ant-..."
python examples/example_usage.py
from src.standard.llm_client import resolve_llm_config, chat_completions

config = {
    "llm": {"provider": "litellm", "model": "anthropic/claude-sonnet-4-6"},
    "api_keys": {},  # litellm reads ANTHROPIC_API_KEY from env
}
llm = resolve_llm_config(config)

result = chat_completions(
    [{"role": "user", "content": "Rewrite this naturally: The experiment was conducted."}],
    api_key=llm["api_key"],
    base_url=llm["base_url"],
    model=llm["model"],
    temperature=llm["temperature"],
    provider=llm["provider"],
)
print(result)
# "We ran the experiment." (or similar humanized output)

Any LiteLLM-supported model string works: openai/gpt-4o, groq/llama-3.3-70b-versatile, deepseek/deepseek-chat, bedrock/anthropic.claude-v2, etc.

@RheagalFire

Copy link
Copy Markdown
Author

cc @molly554

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant