LiteLLM and Model Routing: The Proxy Pattern for Multi-Provider LLM Apps

Every team that moves beyond a single LLM provider hits the same infrastructure problem. You started with OpenAI, then someone wants Claude for long-context tasks, another team spun up a local Ollama instance for privacy-sensitive work, and now your billing is spread across four dashboards, your codebase has four different client libraries, and rotating a compromised API key means touching seventeen files. The proxy pattern solves this. LiteLLM is the most widely adopted open-source implementation of it.

This post covers the full picture: what the proxy pattern actually does, how LiteLLM is architected, how to configure routing and fallbacks, how to run it in production with cost tracking and observability, and an honest comparison against the managed alternatives. If you operate your own AI stack — homelab, startup, or enterprise infra team — this is the pattern you need to understand before you write another provider-specific client.

The Proxy Pattern and Why It Exists

The core idea is simple. You run a service that speaks OpenAI’s API on the inbound side and speaks whatever the target provider requires on the outbound side. Your applications all call the same local endpoint. The gateway handles translation, routing, retry logic, cost accounting, and policy enforcement.

                      ┌─────────────────────────────────────────────────────┐
                      │               Your Applications                     │
                      │                                                     │
                      │  Python app   Node service   curl / cline / aider  │
                      └───────────────────────┬─────────────────────────────┘
                                              │  OpenAI-compatible API
                                              │  POST /v1/chat/completions
                                              │  Authorization: Bearer sk-...
                                              ▼
                      ┌─────────────────────────────────────────────────────┐
                      │                  LiteLLM Proxy                      │
                      │                                                     │
                      │  Virtual Keys    Budget Tracking    Guardrails      │
                      │  Load Balancing  Fallback Chains   Observability    │
                      └─────┬───────────────┬─────────────────┬────────────┘
                            │               │                 │
                   ┌────────▼──┐    ┌───────▼──────┐   ┌─────▼──────────┐
                   │  OpenAI   │    │   Anthropic   │   │  Local Ollama  │
                   │  GPT-4o   │    │  Claude 3.7   │   │  Qwen3-Coder   │
                   └───────────┘    └──────────────┘   └────────────────┘

The benefits compound as you scale. Every API key lives in one place. Every model call is logged in one system. Cost budgets are enforced at the gateway, not in application code. When a provider has an outage, the fallback triggers automatically rather than surfacing a 500 to your users. Adding a new provider — say, swapping one Azure endpoint for another, or adding a regional Bedrock deployment — happens in a config file with zero application changes.

LiteLLM is not the only way to implement this pattern. But it is the one with the most traction in the open-source ecosystem, the widest provider support (100+ LLMs as of 2025), and the most direct path from a single-file Python script to a full production deployment with Postgres, Redis, and a management UI.

Two Interfaces: SDK vs Proxy

LiteLLM ships in two modes that share the same routing and translation logic but serve different integration points.

The Python SDK (pip install litellm) is a drop-in for openai that adds provider translation. You call litellm.completion() with a model string that encodes the provider prefix, and LiteLLM calls the right API:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18

import litellm

# All of these return OpenAI-compatible ChatCompletion objects
resp = litellm.completion(
    model="anthropic/claude-3-5-sonnet-20241022",
    messages=[{"role": "user", "content": "Explain BGP path selection."}]
)

resp = litellm.completion(
    model="bedrock/anthropic.claude-3-5-sonnet-20241022-v2:0",
    messages=[{"role": "user", "content": "Explain BGP path selection."}]
)

resp = litellm.completion(
    model="ollama/qwen2.5-coder:32b",
    messages=[{"role": "user", "content": "Explain BGP path selection."}],
    api_base="http://localhost:11434"
)

Provider-specific parameters (temperature, top_p, system prompts for providers that use a separate system field, etc.) are normalized by LiteLLM before the request goes out. The response object is always the same shape. This is useful for library code or one-off scripts, but it does not give you centralized key management, cross-team cost tracking, or fallback logic that survives a Python process restart.

The Proxy (litellm --config config.yaml) is an actual HTTP server — FastAPI under the hood — that any client can call. Your application does not need to know about LiteLLM at all. You point your OPENAI_BASE_URL to http://litellm-proxy:4000/v1 and everything works. The proxy is the pattern that scales.

Most serious deployments run the proxy. The SDK is useful for experimentation, testing, and embedding in libraries that need portable multi-provider support without requiring a separately deployed service.

Core Configuration: config.yaml

The proxy is driven entirely by a YAML config file. Understanding this file’s structure is understanding LiteLLM.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51

model_list:
  # Each entry defines a model alias that clients request by name,
  # mapped to a specific provider + model + parameters.
  - model_name: gpt-4o
    litellm_params:
      model: openai/gpt-4o
      api_key: os.environ/OPENAI_API_KEY

  - model_name: gpt-4o
    litellm_params:
      model: azure/gpt-4o-2024-11-20
      api_base: os.environ/AZURE_API_BASE
      api_key: os.environ/AZURE_API_KEY
      api_version: "2024-08-01-preview"

  - model_name: claude-sonnet
    litellm_params:
      model: anthropic/claude-3-5-sonnet-20241022
      api_key: os.environ/ANTHROPIC_API_KEY
      max_tokens: 8192

  - model_name: fast-local
    litellm_params:
      model: ollama/qwen2.5-coder:32b
      api_base: http://ollama:11434

litellm_settings:
  # Master key — what clients use to authenticate against the proxy
  master_key: os.environ/LITELLM_MASTER_KEY

  # Callbacks fire on every request/response
  success_callback: ["langfuse", "prometheus"]
  failure_callback: ["langfuse", "sentry"]

  # Default timeout before giving up on a provider
  request_timeout: 300

  # Retry config
  num_retries: 3
  retry_after: 5

router_settings:
  routing_strategy: latency-based-routing
  # Redis required for multi-instance deployments
  redis_host: os.environ/REDIS_HOST
  redis_port: 6379
  redis_password: os.environ/REDIS_PASSWORD

general_settings:
  database_url: os.environ/DATABASE_URL
  store_model_in_db: true

The model_name field is what clients pass to the API. Multiple entries with the same model_name create a pool — requests to gpt-4o above will load-balance between the OpenAI and Azure deployments. This is the core mechanism behind failover and multi-region routing.

Routing Strategies

When multiple deployments share a model_name, LiteLLM picks between them using a routing strategy. The available strategies have meaningfully different trade-offs:

latency-based-routing is the most useful in practice for mixed-provider setups. It maintains a rolling latency average per deployment and weights traffic toward the faster endpoint. If your Azure instance in East US is faster than West US for your users, traffic drifts there automatically without manual tuning.

For purely cost-sensitive workloads — batch processing, evals, non-interactive pipelines — cost-based-routing combined with provider budget caps creates a natural cost ceiling:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25

model_list:
  - model_name: embedder
    litellm_params:
      model: openai/text-embedding-3-small
      api_key: os.environ/OPENAI_API_KEY
      rpm: 3000

  - model_name: embedder
    litellm_params:
      model: azure/text-embedding-3-small
      api_base: os.environ/AZURE_API_BASE
      api_key: os.environ/AZURE_API_KEY
      rpm: 1000

router_settings:
  routing_strategy: cost-based-routing

litellm_settings:
  provider_budget_config:
    openai:
      budget_limit: 50.0   # $50/day max to OpenAI
      time_period: 1d
    azure:
      budget_limit: 30.0   # $30/day max to Azure
      time_period: 1d

When OpenAI hits its $50 daily budget, the router stops sending traffic there and routes entirely to Azure until the window resets. No manual intervention required.

Fallback Chains

Fallbacks are distinct from load balancing. Load balancing distributes successful requests across a pool. Fallbacks activate on failure — 429 rate limits, 5xx errors, timeouts, or content policy rejections.

LiteLLM supports three layers of fallback:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

litellm_settings:
  # 1. Default fallback for any model that gets an error
  fallbacks:
    - {"gpt-4o": ["claude-sonnet", "gemini-pro"]}
    - {"claude-sonnet": ["gpt-4o"]}

  # 2. Context window fallback — when input exceeds model limit
  context_window_fallbacks:
    - {"gpt-4o": ["gpt-4-turbo"]}
    - {"claude-3-5-haiku": ["claude-sonnet"]}

  # 3. Content policy fallback — when provider refuses the request
  content_policy_fallbacks:
    - {"gpt-4o": ["claude-sonnet"]}

The execution path when a primary fails:

Request → gpt-4o (Azure East US)
           │
           ├─ 429 Rate Limit
           │   └─ retry 3x with exponential backoff
           │       └─ still failing
           │           └─ fallback: claude-sonnet
           │               └─ success → return response
           │
           ├─ 413 Context Too Long
           │   └─ context_window_fallback: gpt-4-turbo
           │
           └─ 400 Content Policy
               └─ content_policy_fallback: claude-sonnet

The distinction between these three failure types matters. A rate limit is transient — the model is available, just busy. Retrying with backoff is appropriate before escalating. A context window error is structural — the input is too large for this model, and retrying is pointless; you need a different model. A content policy rejection means the provider is refusing the content entirely, and your fallback provider may have different guardrails.

For order-based priority routing (primary + standby, not a parallel pool), use the order field on deployments:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22

model_list:
  - model_name: production-llm
    litellm_params:
      model: openai/gpt-4o
      api_key: os.environ/OPENAI_API_KEY
    model_info:
      order: 1   # Try first

  - model_name: production-llm
    litellm_params:
      model: azure/gpt-4o-2024-11-20
      api_base: os.environ/AZURE_API_BASE
      api_key: os.environ/AZURE_API_KEY
    model_info:
      order: 2   # Only if order=1 fails

  - model_name: production-llm
    litellm_params:
      model: anthropic/claude-3-5-sonnet-20241022
      api_key: os.environ/ANTHROPIC_API_KEY
    model_info:
      order: 3   # Last resort

This is useful for multi-region deployments where you want your primary region to handle traffic and only failover to a secondary when it’s unavailable.

Virtual Keys and Team Budgets

The virtual key system is what makes LiteLLM usable in multi-team environments. Instead of distributing actual provider API keys, you distribute LiteLLM-issued virtual keys. The proxy is the only service that holds real provider credentials.

Virtual keys are created via the admin API or the web UI:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

# Create a key for the ML team with a monthly budget
curl -X POST http://litellm-proxy:4000/key/generate \
  -H "Authorization: Bearer $LITELLM_MASTER_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "key_alias": "ml-team-prod",
    "team_id": "ml-team",
    "max_budget": 500.0,
    "budget_duration": "monthly",
    "models": ["gpt-4o", "claude-sonnet"],
    "metadata": {"team": "ML", "environment": "production"}
  }'

The response contains a key field starting with sk- — the token your ML team uses in their OPENAI_API_KEY environment variable. It looks identical to an OpenAI key and works the same way from the client’s perspective.

Team budget enforcement works at the proxy level before any provider is called:

Request with virtual key sk-ml-team-xxx
  │
  ├─ Lookup key in Postgres
  ├─ Check team budget: $487.32 spent of $500.00 allowed
  ├─ Budget remaining: $12.68
  ├─ Estimated request cost: $0.18
  ├─ OK — forward to provider
  └─ On response: record spend → $487.50 spent

When a team hits their budget, requests return a 429 with a clear budget-exceeded message rather than a cryptic provider error. You can set soft limits (warn at 80%) and hard limits (block at 100%) independently. Budgets can be reset automatically on a daily, weekly, or monthly cadence.

Teams can also be restricted to specific model pools:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

# In config.yaml
litellm_settings:
  allowed_model_per_team:
    ml-team:
      - "gpt-4o"
      - "claude-sonnet"
    data-team:
      - "embedder"
      - "fast-local"
    security-team:
      - "fast-local"   # air-gapped; local only

This is the same mechanism used to enforce that privacy-sensitive data never leaves your network — route a team to local-only models, and the proxy refuses requests to external providers regardless of what model string the client sends.

Observability

LiteLLM’s callback system fires on every request and response. Callbacks are defined in litellm_settings and receive a full StandardLoggingPayload with timing, tokens, cost, model used, and the actual request/response content.

Prometheus is the lowest-friction integration for teams already running a metrics stack:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

litellm_settings:
  callbacks: ["prometheus"]

# Exposed at /metrics on the proxy port
# Key metrics:
#   litellm_requests_total{model, team, status}
#   litellm_tokens_total{model, team, type}    # type=input|output
#   litellm_spend_total{model, team, provider}
#   litellm_deployment_latency_seconds{model, api_provider}
#   litellm_deployment_success_responses{model, api_provider}
#   litellm_deployment_failure_responses{model, api_provider}

Langfuse captures the full trace — inputs, outputs, latency, cost per call — and is the right choice if you want to debug why a particular generation went wrong:

1
2
3
4
5
6
7
8

litellm_settings:
  success_callback: ["langfuse"]
  failure_callback: ["langfuse"]

environment_variables:
  LANGFUSE_PUBLIC_KEY: os.environ/LANGFUSE_PUBLIC_KEY
  LANGFUSE_SECRET_KEY: os.environ/LANGFUSE_SECRET_KEY
  LANGFUSE_HOST: "https://your-langfuse.example.com"

Every request gets a litellm_call_id that propagates through to Langfuse as the trace ID. You can pass application-level metadata through metadata in the request body and it shows up in Langfuse:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

response = client.chat.completions.create(
    model="claude-sonnet",
    messages=[{"role": "user", "content": prompt}],
    extra_body={
        "metadata": {
            "trace_id": "request-abc123",
            "user_id": "user@example.com",
            "session_id": "session-xyz"
        }
    }
)

OpenTelemetry is the path for teams already invested in Jaeger, Tempo, or a commercial APM:

1
2
3
4
5
6

litellm_settings:
  callbacks: ["otel"]

environment_variables:
  OTEL_ENDPOINT: "http://otel-collector:4317"
  OTEL_HEADERS: "Authorization=Bearer token123"

LiteLLM generates spans for each request including provider roundtrip time, retry attempts, and fallback activations. This makes it possible to see at a glance that your p95 latency spike came from fallback retries to a slow Azure region rather than from application code.

Datadog is supported as a full integration with a custom LiteLLM tile available in the Datadog marketplace, collecting the same Prometheus metrics via the Datadog Agent.

Production Deployment

The minimal production setup requires three components: the LiteLLM proxy, a Postgres database for key/spend persistence, and Redis for distributed rate limiting when running multiple proxy instances.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53

# docker-compose.yml
services:
  litellm:
    image: ghcr.io/berriai/litellm:main-stable
    ports:
      - "4000:4000"
    volumes:
      - ./config.yaml:/app/config.yaml
    environment:
      - DATABASE_URL=postgresql://litellm:${POSTGRES_PASSWORD}@postgres:5432/litellm
      - LITELLM_MASTER_KEY=${LITELLM_MASTER_KEY}
      - OPENAI_API_KEY=${OPENAI_API_KEY}
      - ANTHROPIC_API_KEY=${ANTHROPIC_API_KEY}
      - AZURE_API_KEY=${AZURE_API_KEY}
      - AZURE_API_BASE=${AZURE_API_BASE}
      - REDIS_HOST=redis
      - REDIS_PORT=6379
    command: ["--config", "/app/config.yaml", "--port", "4000", "--num_workers", "8"]
    depends_on:
      postgres:
        condition: service_healthy
      redis:
        condition: service_healthy
    restart: unless-stopped

  postgres:
    image: postgres:16-alpine
    environment:
      - POSTGRES_USER=litellm
      - POSTGRES_PASSWORD=${POSTGRES_PASSWORD}
      - POSTGRES_DB=litellm
    volumes:
      - postgres_data:/var/lib/postgresql/data
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U litellm"]
      interval: 5s
      timeout: 5s
      retries: 10

  redis:
    image: redis:7-alpine
    command: redis-server --requirepass ${REDIS_PASSWORD}
    volumes:
      - redis_data:/data
    healthcheck:
      test: ["CMD", "redis-cli", "--pass", "${REDIS_PASSWORD}", "ping"]
      interval: 5s
      timeout: 3s
      retries: 10

volumes:
  postgres_data:
  redis_data:

For multiple proxy instances behind a load balancer (required for high availability), all instances share the same Postgres and Redis. Redis is what keeps the per-model rpm/tpm counters synchronized across instances — without it, each instance tracks independently and you’ll blow past provider limits.

LiteLLM’s own production documentation recommends at minimum 4 CPU cores and 8 GB RAM for the proxy under meaningful load. The proxy is async (FastAPI + uvicorn workers), so it handles concurrent requests well, but the Postgres connection pool is the typical bottleneck at scale.

For Kubernetes deployments, a Helm chart is available at ghcr.io/berriai/litellm and the standard advice applies: run at least 2 replicas, use a connection pooler (PgBouncer) in front of Postgres if you’re running many replicas, and use a Redis Sentinel or Redis Cluster setup rather than a single Redis instance.

                    ┌─────────────────┐
                    │   Load Balancer │
                    │  (nginx/Traefik)│
                    └────────┬────────┘
                             │
              ┌──────────────┼──────────────┐
              │              │              │
         ┌────▼───┐    ┌────▼───┐    ┌────▼───┐
         │LiteLLM │    │LiteLLM │    │LiteLLM │
         │  :4000 │    │  :4000 │    │  :4000 │
         └────┬───┘    └────┬───┘    └────┬───┘
              └──────────┬──┘─────────────┘
                    ┌────▼────┐
                    │ Redis   │  ← rpm/tpm sync across instances
                    └─────────┘
                    ┌─────────┐
                    │Postgres │  ← keys, teams, spend, audit log
                    └─────────┘

Guardrails

LiteLLM supports content filtering and PII detection as middleware that runs in the proxy before forwarding to providers or after receiving responses. Guardrails integrate with external systems (Presidio, Lakera Guard, Aporia, AWS Bedrock Guardrails) and can also be custom Python callbacks.

A basic PII detection setup using Presidio:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

guardrails:
  - guardrail_name: pii-masking
    litellm_params:
      guardrail: presidio
      mode: pre_call   # scan input before sending to provider
      output_parse_pii: true   # also scan responses

  - guardrail_name: prompt-injection
    litellm_params:
      guardrail: lakera_prompt_injection
      api_key: os.environ/LAKERA_API_KEY
      mode: pre_call

Guardrails can be applied globally, per-team, or per-key. A team handling regulated data can have mandatory PII masking while an internal tooling team has none, all through the same proxy without any application code changes.

Custom guardrails are Python classes with async_pre_call_hook and async_post_call_success_hook methods:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15

from litellm.integrations.custom_guardrail import CustomGuardrail
from litellm.types.guardrails import GuardrailEventHooks

class ConfidentialProjectBlocker(CustomGuardrail):
    def __init__(self):
        self.blocked_patterns = ["Project Nightingale", "Operation Coldwater"]
        super().__init__(supported_event_hooks=[GuardrailEventHooks.pre_call])

    async def async_pre_call_hook(self, user_api_key_dict, cache, data, call_type):
        messages = data.get("messages", [])
        for msg in messages:
            content = msg.get("content", "")
            if any(p in content for p in self.blocked_patterns):
                raise ValueError(f"Request contains blocked content")
        return data

Register it in config.yaml:

1
2

litellm_settings:
  callbacks: ["mymodule.ConfidentialProjectBlocker"]

Auto Routing and Complexity Classification

LiteLLM introduced auto routing as a way to automatically select a model based on the estimated complexity of the incoming request, rather than requiring callers to specify a model tier. A complexity router uses a fast classifier model to bucket requests into simple/medium/complex and routes each bucket to a different deployment.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18

router_settings:
  routing_strategy: auto

model_list:
  - model_name: simple-task
    litellm_params:
      model: openai/gpt-4o-mini
      api_key: os.environ/OPENAI_API_KEY

  - model_name: complex-task
    litellm_params:
      model: anthropic/claude-opus-4-7
      api_key: os.environ/ANTHROPIC_API_KEY

litellm_settings:
  auto_routing_model_tiers:
    simple: simple-task
    complex: complex-task

The appeal is cost reduction without application logic: a batch job that generates summaries routes short inputs to gpt-4o-mini at $0.15/1M tokens and only escalates to Claude Opus for genuinely complex reasoning tasks. In practice, the classification is imperfect — complexity classifiers struggle with domain-specific difficulty and tend to over-promote short but semantically dense requests. Treat this as a heuristic optimization, not a reliable quality gate. If output quality matters, explicit model selection from the application is more predictable.

Tag-Based Routing

Tags let you route specific requests to specific deployment pools without requiring callers to know deployment details:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

model_list:
  - model_name: llm
    litellm_params:
      model: openai/gpt-4o
      api_key: os.environ/OPENAI_API_KEY
    model_info:
      tags: ["public", "external"]

  - model_name: llm
    litellm_params:
      model: ollama/qwen2.5-coder:32b
      api_base: http://ollama:11434
    model_info:
      tags: ["private", "internal"]

Callers specify tags in the request body:

1
2
3
4
5

response = client.chat.completions.create(
    model="llm",
    messages=[{"role": "user", "content": prompt}],
    extra_body={"tags": ["private"]}  # routes only to Ollama
)

Combined with virtual key restrictions per team, this becomes a data residency enforcement mechanism: keys issued to teams handling PII have tags: ["private"] baked into their configuration, and the proxy refuses any request from those keys that would route to an external provider.

Comparison: LiteLLM vs Managed Alternatives

The LLM gateway space has consolidated around three distinct positions:

OpenRouter wins on time-to-working. One API key, access to 200+ models, zero infrastructure. The 5.5% fee is negligible at low spend but becomes meaningful at scale — at $10K/month API spend that’s $550/month in fees. There are also no virtual keys, no team budgets, and no on-premises data residency option.

Portkey is the best option if you want enterprise observability features (semantic caching, advanced tracing, managed guardrail policies) without operating your own infra. The managed tier keeps your data off your servers, which is a disqualifier for compliance-sensitive workloads. The self-hosted Enterprise tier addresses this but pricing is opaque.

LiteLLM is the choice when you need data residency, custom guardrails, integration with existing Prometheus/Langfuse/OTEL infrastructure, or team budget enforcement that your finance team can actually see in a dashboard. The operational cost is real: you maintain the deployment, the Postgres instance, and the Redis cluster. But for organizations that already operate Kubernetes or Docker Swarm stacks, this is a solved problem.

Common Pitfalls

Not using Redis for multi-instance deployments. Without Redis, rpm/tpm counters are per-instance. Three proxy replicas each think they’re allowed 1000 rpm to OpenAI, so your actual combined rate is 3000 rpm — until OpenAI 429s all of them simultaneously.

Forgetting context window fallbacks. Provider rate limits get a lot of attention, but input truncation is equally disruptive. A request that exceeds gpt-4o’s 128K context window returns a 400, not a 429, and the default fallback logic doesn’t catch it unless you explicitly configure context_window_fallbacks. Long document processing pipelines that don’t set this will fail silently on larger inputs.

Using master key in application code. The master key has full admin access to the proxy — it can create and delete other keys, access all logs, and reconfigure the proxy. Application workloads should always use virtual keys with scoped permissions and budgets.

Treating model_name as opaque. Clients pass model_name to the proxy, but some clients also send this string to the provider directly in error reporting or tracing. Using provider-specific names (gpt-4o, claude-3-5-sonnet-20241022) as your model_name means your abstraction leaks into observability data and you lose the ability to silently swap providers without client changes.

Ignoring the warm-up period for latency-based routing. On startup, all deployments have equal latency estimates. Traffic distributes evenly until enough requests accumulate to build accurate estimates. If you’re doing load testing immediately after deployment, the first few minutes of results reflect random distribution, not the steady-state routing behavior.

Not pinning the image tag. litellm:main-stable tracks the release branch and can introduce breaking config changes between updates. For production, pin to a specific version tag (ghcr.io/berriai/litellm:v1.x.y) and upgrade deliberately.

Integrating with Existing Tools

Because the proxy is OpenAI-compatible, any tool that accepts OPENAI_BASE_URL and OPENAI_API_KEY works without modification:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20

# Aider
export OPENAI_BASE_URL=http://litellm-proxy:4000/v1
export OPENAI_API_KEY=sk-virtual-key-ml-team

# Claude Code
export ANTHROPIC_BASE_URL=http://litellm-proxy:4000

# LangChain
from langchain_openai import ChatOpenAI
llm = ChatOpenAI(
    base_url="http://litellm-proxy:4000/v1",
    api_key="sk-virtual-key-ml-team",
    model="claude-sonnet"  # your model alias
)

# curl
curl http://litellm-proxy:4000/v1/chat/completions \
  -H "Authorization: Bearer sk-virtual-key-ml-team" \
  -H "Content-Type: application/json" \
  -d '{"model": "gpt-4o", "messages": [{"role":"user","content":"test"}]}'

The proxy also exposes /v1/models which returns all configured model aliases. Tools that enumerate available models will see your aliases rather than provider-specific model IDs, which is typically what you want.

The Honest Trade-off

LiteLLM is genuinely useful infrastructure for any team running more than one LLM provider, and the proxy pattern is the right abstraction for this problem. But it adds a service to operate. That service has a database, a cache, and real upgrade cadence — the project moves fast and config schema changes between versions.

For a solo developer experimenting with multiple providers, the SDK mode or OpenRouter is probably the right starting point. For a team of five shipping a product, the overhead of a self-hosted proxy starts to pay off in reduced operational toil around key management and cost visibility. For a twenty-person team with a dedicated platform function, LiteLLM becomes a foundational piece of infrastructure alongside your CI/CD and logging stack.

The question to answer before deploying it: do you need the things the gateway provides — centralized keys, team budgets, provider-agnostic clients, observability across providers — or do you just need a quick way to call multiple APIs? The first case is a gateway problem. The second case is probably just an environment variable.