Originally published on kuryzhev.cloud
Your Loki dashboard shows the raw nginx access log fine — but every label query returns zero results, and nobody notices until the ingesters start OOMing. We hit this exact scenario on a client's edge cluster: {job="nginx"} |= "500" in Grafana Explore returned every 500 error you'd expect, but {job="nginx", status="500"} came back empty. That gap — raw text works, labels don't — is almost always a broken loki regex pipeline stage, and it's one of the most common Promtail/Alloy misconfigurations we see in the field.
Symptoms
The failure mode is deceptively quiet. Here's what we actually saw before we understood what was happening:
LogQL queries using extracted labels like {job="nginx", status="500"} return zero results, even though {job="nginx"} |= "500" shows plenty of matching raw lines.
