Our team used multiple LLM providers from different clouds: Google, Azure, and AWS. Every app and project had its own API key. We could not see which key belonged to which app, team, env etc. We also had no clear view of token usage and tracking. Later, finance asked which product spent a lot of money. We could not answer.
First I tried LiteLLM. It is a good tool if you need to quickly setup your llm gateway. Then the team became larger. LiteLLM Admin UI SSO is free for only 5 users. After 5 users, you need the Enterprise plan. Org admin and SCIM are also Enterprise. I did not want a license wall on the control plane.
So I built Open LLM Gateway and released the full stack as Apache 2.0. That includes the admin UI, not only the proxy.
What is in the repo:
OpenAI /openai/* and Anthropic /anthropic/* proxy
