Un-Blackboxing vLLM: Building an AI SRE Copilot with SigNoz
When moving from external APIs (like OpenAI) to self-hosted open-source models, developers hit a wall: AI infrastructure is a black box. If an AI Agent hallucinates or costs explode, standard CPU metrics won’t tell you why.
For the Agents of SigNoz Hackathon (Track 1), I solved this by transforming a Kaggle T4 GPU running vLLM into an enterprise-ready system. I built an SRE Copilot and FinOps Gateway, but the real superhero of this architecture is SigNoz Cloud.
Here is how I used OpenTelemetry and SigNoz to make sense of complex AI data.
To treat AI like true enterprise infrastructure, I built three layers of custom observability in Python:
