LLM gateway: OpenRouter, LiteLLM or build your own?

In June, we wrote on LinkedIn (in French) that we were worried about companies that had built their agents around a single provider. In two months, Anthropic's premium model had gone from $5/$25 to $10/$50 per million tokens, right after a tokenizer change that already made the same request consume 30 to 45% more tokens, according to OpenRouter's measurements. A bill multiplied by 2.6 to 2.9, without a single line of code changed.

Their model was doing the job just fine. Their mistake was elsewhere: the application called a single API, hard-coded. When the provider goes down, raises its prices or retires a model, all you can do is take it.

The layer that keeps you out of that corner is called an LLM gateway, or AI gateway. The market offers every shape of it, from OpenRouter, the best-known service, to LiteLLM, the open-source reference. People usually compare them on catalog size and price. For a company, we start with a different question: where does your data go?

What is an LLM gateway, or AI gateway?

An LLM gateway is a software layer that sits between your applications and the providers of AI models. It exposes a single API, sends each request to the most suitable model and provider, and automatically switches to another one when the first fails.

It runs no model itself. That is what sets it apart from an inference provider, which runs models on its own GPUs. We cover that difference in our guide to AI inference.

Several names cover the same thing: AI gateway, LLM gateway, multi-provider gateway. The term LLM router sometimes refers to a narrower building block, which picks the model based on how hard the question is, without handling fallback or keys.

What a gateway actually does

It fails over when a provider goes down

No inference provider holds 99.9% availability on its own. Here is what we measured in production on five of them, over thirty days in July 2026:

Success rate by inference provider: Scaleway 99.87%, Nebius 99.81%, Cerebras 99.78%, DeepInfra 99.76%, Mistral 99.56%

Taken one by one, none reaches three nines. Stacked behind a gateway that checks their health and switches as soon as one falls behind, you get there. The full reasoning, with the contractual commitments of the major providers, is in our article on enterprise AI SLAs.

Failover assumes the same model runs at several providers, or that you accept a fallback model. That is one of the concrete benefits of open-weight models: you change provider without touching your prompts.

It arbitrates between price, speed and country

This is the least-known function, and the one that pays off most. The same model does not have the same price, the same speed or the same nationality depending on who serves it. Our readings:

Two bar charts. On the left, the gpt-oss-120b model: Nebius serves it at 40 tokens per second and Groq at 500, both at $0.15 input and $0.60 output per million tokens, Cerebras at 3,000 tokens per second for $0.35 and $0.75. On the right, the DeepSeek V4 Pro model costs $0.57 per million tokens at DeepSeek (China), $1.69 at DeepInfra (United States), $2.11 at Umans and $2.21 at Arkane Cloud (France), $2.27 at Nebius (Netherlands).

On the left, gpt-oss-120b. Nebius and Groq charge the same price for it, $0.15 input and $0.60 output per million tokens. The first serves it at 40 tokens per second, the second at 500. Cerebras goes up to 3,000 tokens per second for a barely higher rate. A 75x difference in speed, at almost the same price.

On the right, DeepSeek V4 Pro. Served by DeepSeek, it costs $0.57 per million tokens. At Nebius, $2.27. The same model, four times more expensive. But the cheapest is a Chinese company, and the three most expensive are European. The lowest price is not always the one you are allowed to pay.

One last subtlety, invisible on the chart: the flag shows the country of the legal entity, not of the data center. Nebius, a Dutch company, serves gpt-oss-120b from Finland, but DeepSeek V4 Pro from the United Kingdom.

A gateway that reads these three variables picks, request by request, the combination that meets your constraint at the best cost.

It centralizes keys and accounts

The application only knows one API, usually compatible with OpenAI's. Provider keys are stored in one place. Changing models becomes a line of configuration rather than a project, and that is what makes trade-offs possible like the one in our OCR article, where an open-weight model cut the bill by 10x.

The downside is immediate: the gateway holds all your keys. We come back to this below, with LiteLLM.

It measures consumption

Cost per team, per application or per user, quotas, call logs. This is often the function that wins over the finance department, well before availability does.

The five families of gateways

How to read: green is favorable, red unfavorable. The ± sign marks a criterion that depends on the vendor or on your configuration.

The sovereignty column only makes sense next to the previous one. A gateway hosted on your own servers in Europe can very well send your requests to the United States if nobody configured it not to.

OpenRouter: what it is, and what to check before using it in a company

OpenRouter is an American SaaS gateway: a single API, compatible with OpenAI's, that gives access to several hundred models served by dozens of providers. You buy credits, you call a model, OpenRouter picks the provider and fails over when something breaks. The company raised $113 million in May 2026, in a round led by CapitalG, Alphabet's growth fund (Dealroom).

For a developer who wants to try ten models in an afternoon, it is unbeatable. For a company, four points deserve a check before any data goes through it.

The business model first. OpenRouter passes provider prices through without markup and makes its money on credit purchases: a 5.5% fee, with a $0.80 minimum (OpenRouter FAQ). If you bring your own provider keys, the first $25,000 of monthly usage carries no fee, then OpenRouter takes 5%.

On data, OpenRouter does not store prompts or responses by default. Two options change that, if you turn them on: logging of your exchanges, and permission for OpenRouter to use them, in exchange for a 1% discount (data collection policy). What matters happens next, at each provider called, which applies its own policy.

Routing control is OpenRouter's real strength, provided you configure it. The data_collection: "deny" parameter excludes providers that collect data, zdr: true restricts routing to providers with zero data retention, and the only and ignore lists set who is allowed to answer (provider selection). The documentation itself warns that every restriction reduces fallback options.

Finally, jurisdiction. OpenRouter offers EU-only routing, through eu.openrouter.ai, reserved for its Business and Enterprise plans. Processing then stays within the European Union. The entity, however, is American, and therefore subject to the CLOUD Act like any American provider, wherever its servers run. We explain why this matters in What if the United States Turned Off Your Digital Tap?

A word on free models, which attract a lot of curious users. They are limited to 50 requests per day, or 1,000 after buying $10 of credits. Perfect for testing, unusable in production.

What is the European alternative to OpenRouter?

On the SaaS side, Eden AI, a French vendor, offers a European endpoint that only routes to providers and models cleared for processing in Europe, and states that it keeps neither prompts, files nor outputs (Eden AI documentation). As with OpenRouter, the guarantee applies to a model and provider pair: check it for each one you enable.

The other option is to host an open-source gateway yourself in a European data center. The most widely used is LiteLLM.

LiteLLM: the open-source gateway you host yourself

LiteLLM is the open-source reference in the field: a Python library and a proxy server that expose an OpenAI-compatible API in front of more than a hundred providers. You host it wherever you want, including in Europe, and your requests go through no third party before reaching the final provider.

What the tool does not do for you is run it. You have to update it, make the gateway itself highly available (otherwise it becomes the single point of failure it was meant to remove), and above all measure. Knowing which provider answers fast right now takes probes, thresholds and history. It is the part we underestimated most when building our own gateway.

Then there is security. On March 24, 2026, two compromised versions of LiteLLM, 1.82.7 and 1.82.8, were published on PyPI. They contained a credential stealer that harvested environment variables, SSH keys, cloud credentials and database passwords (LiteLLM security advisory). They stayed online for about forty minutes. The official proxy Docker image, which pins its dependencies, was not affected, and a clean release came out within the week. Searches for LiteLLM more than tripled in France that month.

The lesson goes beyond LiteLLM. By design, a gateway holds the keys to all your providers: it is the most profitable target in your AI infrastructure. If you host one, pin your versions, isolate it and treat it as a vault rather than a utility.

A gateway is also a sovereignty tool

The gateway is usually presented as a reliability tool. It is also the only place in your architecture where a sovereignty rule applies to every request at once.

"No data goes to a non-European provider." Without a gateway, that rule depends on the discipline of every developer who writes an API call. With a gateway, it is a line of configuration, and automatic failover can no longer send you, on an outage night, to a provider you never approved.

Three conditions for the rule to hold:

1. The gateway itself must be European. Otherwise, all your requests go through an entity subject to the CLOUD Act before reaching Mistral or Scaleway.
2. The filter must cover both the country of the entity and that of the data center. The Nebius case above shows it: a European company can serve a model from outside the Union.
3. You need enough European providers to fail over to. Every provider removed from the list is one fewer backup route: that is the price of sovereignty, and you are better off measuring it before the first outage.

The criteria we apply to any AI vendor are detailed on our sovereign AI page.

Do you need a gateway?

Not yet if you are in a POC, with one model, one use case and no availability commitment. Still, write your calls behind a thin abstraction layer: the day you need a gateway, you will plug it in without rewriting the application.

Yes as soon as an application goes into production with users who depend on it, as soon as several teams consume models, or as soon as a sovereignty constraint has to apply to every call.

And it is not yours to build if you are buying an AI platform that includes one. Just ask the vendor to describe it: which providers, in which countries, with what failover rules and what no-training commitment.

At Ask This Guy

The gateway is built into the platform: our customers have nothing to build or operate. It continuously selects the models that answer, and answer fast, with a service target above 99.9% on the defined scope.

The sovereignty rule is an organization setting. EU only: Mistral, Scaleway, Nebius, Black Forest Labs, plus our own GPUs at Verda, in Finland and Iceland. Worldwide: the same, plus selected non-European providers. The setting can be changed at any time from the console, and the details are public on our AI provider management page.

Model costs are included in the subscription. The price and speed gaps described above are our problem, not yours.

And if you already have your own gateway or a contract with a provider, our Bring Your Own AI mode lets you plug them into the platform. It is available on request.

Frequently asked questions about LLM gateways

LLM gateway or AI gateway: what is the difference?

Both terms describe the same layer: a single entry point in front of several model providers, with automatic failover. AI gateway is slightly broader, since some products also cover image, audio or embedding models. LLM gateway stresses language models, which make up most of the traffic in a company.

What is the difference between an LLM gateway and a regular API gateway?

A regular API gateway routes, authenticates and rate-limits calls to your own services. An LLM gateway does the same job in front of AI model providers, and adds what this field requires: counting tokens, comparing the price and speed of several providers for the same model, and failing over to another provider or model when one breaks. Several API gateways on the market, such as Kong, now offer a dedicated AI module.

What is OpenRouter?

OpenRouter is an AI gateway sold as SaaS by an American company. It gives access to several hundred models, from OpenAI, Anthropic, Google, Mistral or open-weight families, through a single OpenAI-compatible API. It picks the provider that serves each request, fails over when one breaks, and makes its money through a 5.5% fee on credit purchases.

Is OpenRouter GDPR compliant?

OpenRouter does not store prompts by default, lets you restrict routing to providers with zero data retention and offers EU-only processing on its Business and Enterprise plans. The company remains American, and therefore subject to the CLOUD Act. Compliance mostly depends on your configuration and on the final providers called: it has to be checked case by case with your DPO.

LiteLLM or OpenRouter: which one should you choose?

OpenRouter is a turnkey service: nothing to host, but your requests go through an American company that takes 5.5% on credits. LiteLLM is open-source software you host yourself: you choose where your data transits, but you run, secure and update it. The first suits prototyping, the second suits teams that have infrastructure and a sovereignty constraint.

Does a gateway add latency?

A little, but rarely noticeably. The detour through a gateway costs a few tens of milliseconds at most, while generating an answer takes seconds. A gateway that picks the fastest provider saves far more time than it costs: on gpt-oss-120b, the gap between providers ranges from 40 to 3,000 tokens per second.

Can you use OpenRouter's free models in a company?

For testing, yes. For production, no: they are limited to 50 requests per day, or 1,000 after buying $10 of credits, with no service commitment. And a model served only by providers that retain data is only available if you have allowed those providers in your settings: check them before sending anything.

---

Conclusion

An application wired to a single provider is stuck with its outages, its price increases and its jurisdiction. A gateway gives you those three decisions back, provided you know where it runs and who controls it.

If you want to see a gateway enforce a European policy under real conditions: book a demo.