LLM infrastructure · September 9, 2026
Open Source LLM Gateways: Library or Proxy?
LiteLLM, Portkey, Bifrost and freelm all remove the hosted router from your request path. They are not the same shape, and the shape decides more than the feature list.
By Shihab Shahriar Antor · Updated 2026-09-09
The reason to want one of these is usually specific: a hosted router like OpenRouter is a third party in the request path, and either the latency, the margin, or the data handling has become a problem. Removing it means running the routing yourself.
Every project in this space gets compared on provider count, which is the least useful axis. The decision that actually constrains you is architectural, and it comes first.
The two shapes
| In-process library | Standalone proxy | |
|---|---|---|
| Network hops added | None. Your code calls the provider directly. | One, to a service you run. Usually a local network hop, so small but not zero. |
| Language reach | Only the language it is written in. A Go service cannot use a Python library. | Every language, because the interface is HTTP. Polyglot stacks effectively require this shape. |
| Where keys live | In every service that calls a model. Wider blast radius, more places to rotate. | One place. Application code holds a gateway credential rather than provider keys. |
| Operational load | A dependency version. Nothing to deploy, monitor or scale. | A service to deploy, monitor and scale, and a new single point of failure to make redundant. |
| Central logging and spend limits | Per-service, so you assemble the picture yourself. | Built in, since every request passes one point. This is often the real reason teams adopt one. |
Some projects ship both shapes. LiteLLM is used as a Python SDK and as a proxy server, so picking it does not settle the question on its own.
The projects
Four worth knowing, with what each is actually for. All are open source and all remove the hosted router from the path.
| Project | Shape | Written in | Best fit |
|---|---|---|---|
| LiteLLM | SDK and proxy server | Python | The default choice, and the one with the widest provider coverage by a large margin. Pick it unless you have a specific reason not to. |
| Portkey Gateway | Proxy | TypeScript | Edge-deployable gateway with routing, retries and observability. Good when you want the proxy shape close to your users rather than in one region. |
| Bifrost | Proxy | Go | Built for throughput. Worth evaluating when the gateway's own overhead is measurable in your latency budget, which is a real problem at volume and imaginary below it. |
| freelm | Library | Python and JavaScript | Narrow on purpose: routing and failover across free-tier providers, with paid models refused unless explicitly requested. Ours. |
Coverage and features move quickly in this category. Check each project's own README before committing rather than trusting any table, including this one.
What the feature lists leave out
Four things decide whether one of these works in production, and none of them appear on a comparison page.
- 01
Does failover survive streaming
Switching providers mid-stream is genuinely hard, so it is often the case that is quietly unsupported. If your product is a chat interface, failover that only covers non-streaming calls is failover you cannot use.
- 02
Is quota state tracked locally
A router that learns it is rate limited by being rate limited pays a wasted round trip on every discovery, and adds that latency to whatever is queued behind it. Local state is the difference between graceful degradation and a slow outage.
- 03
Are candidates interleaved across providers
Order the fallback list naively and one provider with a long model catalogue consumes every retry attempt before the router reaches a healthy alternative. This one is easy to miss until the day a provider goes down.
- 04
Can the model list change without a deploy
Provider catalogues turn over constantly. A hardcoded model list is a scheduled outage on whatever day a model is deprecated, so runtime discovery matters more than it sounds.
Honest placement of ours
freelm is a library, not a proxy, and it is deliberately narrow. It routes across six providers that serve models at no cost, refuses paid identifiers unless you ask for them, tracks quota locally, opens a circuit breaker on repeated failure, and keeps streaming working across a switch. It is MIT licensed on PyPI and npm.
If you are routing paid production traffic across many vendors, LiteLLM is the better tool and it is not close. freelm exists for the case where the goal is running on free capacity without that meaning a single provider you cannot depend on. The routing design is written up in routing across free LLM providers, and the provider comparison is in OpenRouter alternatives with a real free tier.
Questions
- What is an LLM gateway?
- A layer between your application and model providers that handles one API surface across vendors, retries and fallback when a provider fails, key management, and usually logging and spend limits. It exists so that switching or adding a model provider is a configuration change rather than a code change.
- Is there an open source alternative to OpenRouter?
- Yes, though the comparison is not one to one. OpenRouter is a hosted service that also sells you the capacity. Open source gateways give you the routing and leave you to bring your own provider keys and pay each provider directly. LiteLLM is the closest in coverage; Portkey Gateway and Bifrost are proxies; freelm is a library focused on free tiers.
- Should I self-host an LLM gateway or use a hosted router?
- Self-host when a third party in the request path is a problem, which usually means data handling constraints, latency budgets, or wanting to avoid the reseller margin. Use a hosted router when you would rather not operate the thing, and the margin is cheaper than the engineering time. Neither is the default correct answer.
- Does a gateway add latency?
- A library adds effectively none, since it runs in your process and the request still goes straight to the provider. A proxy adds one hop, typically small on a local network and larger if it sits in another region. Both are small next to model inference time, but they are not free and they are worth measuring rather than assuming.
- LiteLLM or Portkey?
- Different shapes, so start there. LiteLLM is a Python project used as an SDK or a proxy, with the widest provider coverage available. Portkey Gateway is a TypeScript proxy designed to be deployed at the edge. For a Python codebase wanting a library, LiteLLM. For a polyglot stack wanting a gateway close to users, Portkey.