The 40KB Problem Nobody Noticed Until Cloudflare Billed Us
Every viral video we ingest at ViralVidVault crosses three service boundaries before it ever reaches a user. A PHP 8.4 ingestion worker pulls the raw metadata, hands it to a Go trend-scoring service, which in turn feeds a Python analytics pipeline that computes velocity and acceleration curves for the European feeds. For a long time these three services spoke JSON to each other, because JSON is what everyone reaches for first. It worked, it was debuggable, and nobody questioned it.
Then we looked at the numbers. A single enriched VideoMetadata record — title, tags, region, published timestamp, view counts, and a nested block of trend signals — averaged just under 40KB as pretty-printed JSON, and around 28KB minified. Multiply that by the ~2.1 million records that move between services every day during a trend spike, and the internal egress alone was measurably showing up on our Cloudflare bill. Worse, the Go service was burning real CPU on encoding/json reflection, and the PHP worker spent more time in json_encode than it did doing the actual HTTP fetch.
We migrated the inter-service contract to Protocol Buffers. Payloads dropped by roughly 68%, parse CPU on the Go side fell by more than half, and — the part I care about most as someone shipping under GDPR — the schema became a single enforced contract instead of a loose bag of keys. This is the write-up I wish I'd had before starting. If you want to see the end result in production, it's the discovery engine behind ViralVidVault, our GDPR-compliant European viral video tracker.
