Every time I put a model behind an endpoint I make the same lazy decision. I pick whatever I used last time, or whatever I read about most recently, and I tell myself I'll benchmark it properly later, and later never arrives because there is always something with an actual deadline on it and comparing model latencies feels like procrastination even when it isn't. I never do it. Not once.
So I built the thing that would make me do it. One prompt, fired at six models at once, streaming side by side in columns, with time to first token and cost per run underneath each one. About 390 lines of Python. Code's here, MIT, take it.
Then I ran it, and three things happened that I didn't plan for.
The integration is two lines, and that's the least interesting part
DigitalOcean's inference endpoint speaks OpenAI, so this is the whole thing:
