# LLM API Routing Layer

A routing layer sits between an application and multiple LLM providers, exposing one OpenAI-compatible endpoint and managing the provider matrix below. The motivation is operational: model quality, pricing, and uptime move faster than typical product release cycles, so applications that hard-code a provider take on volatility risk that has nothing to do with their actual product surface.

What a routing layer typically provides:
- **Provider abstraction** — same endpoint regardless of which provider serves the request
- **Fallback chains** — primary fails (5xx, latency anomaly, rate limit) → secondary takes over without app-side handling
- **Routing rules** — pick a model based on request characteristics (token count, complexity heuristics, user tier, cost ceiling)
- **Unified billing** — one invoice across providers
- **Observability** — uniform logs and metrics across providers
- **Optional features** — prompt caching, semantic dedup, response caching, KV reuse

Trade-offs:
- Adds a hop (latency overhead, single point of failure)
- Trust the router with prompts and completions (privacy posture matters)
- Pricing model — most charge a markup; some (like [[toolbox/routing-run|routing.run]]) charge per-request rather than per-token, which inverts the math at large context sizes

Notable examples:
- **OpenRouter** — the dominant aggregator; per-million-token markup; supports almost every public model
- **Chutes** — similar model
- **routing.run** — per-request pricing, claims zero logging, open-source routing infra
- **Pi** ([[lucumr-blog]]) — embeds provider selection into a coding agent rather than as a separate gateway
- **Self-hosted (LiteLLM, Helicone, Portkey)** — routing as a library/middleware you run yourself

Adjacent: this is structurally similar to the [[alternative-frontend-pattern]] for service consumption — a third-party UI/proxy that improves on the underlying API contract — but with stronger lock-in implications because the routing layer mediates billing as well as routing.

The pattern's main long-term risk is that it depends on continued provider neutrality. If one provider starts requiring direct integration for advanced features (longer contexts, specific tool-use formats, fine-tuning), routers fall behind unless they negotiate special access.

Cross-references: [[toolbox/routing-run]], [[lucumr-blog]].