How we added native Spanner support to Kubernetes autoscaling — and what we learned along the way
The problem
We run several workloads on Kubernetes that process jobs stored in Cloud Spanner tables. The pattern is simple: a producer writes rows with status = 'pending', workers pick them up and mark them done. The question is — how many workers do you run?
Fixed replica counts mean either wasted money during quiet periods or dropped throughput during spikes. We needed autoscaling based on actual queue depth, not CPU or memory.
KEDA (Kubernetes Event-Driven Autoscaling) is the standard answer for this — it scales workloads based on external metrics like queue lengths, database counts, and custom queries. It already had scalers for GCP Pub/Sub, Cloud Tasks, and Cloud Storage. But not Spanner.
