A Caddy Cert Expired Because systemd-resolved Was Selectively Lying

Here's something that took longer to debug than it should have.

The setup

Running Caddy as a reverse proxy on a systemd-based Linux machine. Cert renewal via ACME. Everything looks fine in the logs. Then one day the cert is expired and nobody noticed for two days.

The cause

systemd-resolved has a behavior where it returns SERVFAIL for specific DNS queries depending on the upstream resolver situation. It's not consistent. Some zones resolve fine. Some silently fail. Caddy's ACME client sends the challenge request, systemd-resolved reports a failure, and the renewal just... doesn't happen.