Most CI pipelines assume a function called with the same input twice returns the same output. That assumption breaks the moment an LLM call enters your test suite. Ask GPT-4 or Claude the same question twice and you can get two different (both correct) answers. Teams shipping LLM-backed features often respond to this by writing almost no tests for the LLM-touching code paths, or by asserting on exact string output and then disabling the test the first time it flakes. Neither is sustainable once the feature is in production and a prompt change or model upgrade can silently break behavior.
The fix isn't a clever assertion library. It's splitting what you're actually testing into three tiers with different determinism guarantees, and mapping each tier to a different CI job.
Tier 1: Contract tests (deterministic, run on every PR)
A contract test never calls a live model. It asserts on the shape of what your pipeline produces, not the content. If your orchestrator expects the LLM to return {title, tags, price_usd, full_content}, a contract test feeds a canned response through your parsing/validation layer and checks it doesn't throw, that required fields are present, and that types match.
# test_contract.py
