Every LLM endpoint's landing page says fast, reliable, compatible, secure. Those words are free. This is the list of questions that aren't free — the ones where the answer takes a vendor real work to be able to give, and where you can verify the answer yourself in an afternoon.
Disclosure up front: I work on daoxe, a multi-model gateway. At the end I run this checklist against it, and two rows don't pass. That's the point of publishing it — a checklist that flatters whoever wrote it isn't a checklist, it's a spec sheet.
1. Protocol coverage — native or translated?
Ask which wire protocols the endpoint implements and which ones it translates into. These are different products. A translation layer is fine for text chat and lossy for agents: tool-call block ordering, streamed tool arguments, prompt-cache markers and stop-reason fidelity all live in fields that have no equivalent on the other side.
Evidence: for the Anthropic protocol, omit the required max_tokens from a /v1/messages call. A native implementation returns 400; a translation layer often fills in a default and returns 200. For the OpenAI protocol, check whether streamed tool calls arrive as incremental fragments with an index, or all at once at the end.
