OmniRoute
One endpoint for every model, with fallbacks when a provider goes down.
- TypeScript
- LLM routing
- Self-hosting
Mission
Keep the Lights On
Providers fail at random. Order your fallback chain and pick a compression mode to keep uptime above 99% without burning money.
Best: not set yet
Skip to the write-upEverything in the game is made-up data. Press Escape any time to pause.
Why I use it
OmniRoute is an open-source AI gateway: one endpoint in front of 160+ providers, with automatic fallback. I use it every day for my coding agents, so a rate limit or an outage at one provider does not stop my work. The project is not mine, I run my own fork and send fixes upstream.
How I run it
I host it on a small cloud VM and reach it over Tailscale. That box has 1 GB of memory, so I cap the service and give it swap.
One fix I sent upstream came from the model picker in my coding agent. The gateway could route to reasoning-effort variants of a model, but its model list never advertised them, so clients that only read the list could not see them.
What surprised me
I expected prompt compression to be where the savings came from. It was not. On code-heavy traffic the gateway checks whether compression would corrupt the content and reverts it, and that check fired almost every time. The real savings came from the provider prompt cache. Level 3 of the game is built around that.