Most data pipelines are built around a comfortable assumption: the data will show up roughly when you expect it, in roughly the format you expect, and if something breaks, you can reprocess yesterday's batch and call it a day.
Space science infrastructure exists to destroy that assumption.
When engineers at NASA built the Ziggy framework to handle science data from the Kepler and TESS missions, they weren't solving a niche astronomy problem. They were solving one of the hardest general-purpose pipeline problems that exists: how do you build a system that orchestrates multi-language, multi-stage processing across heterogeneous compute environments, where the data provenance has to be airtight and the cost of losing a record is genuinely catastrophic?
The answers they arrived at are worth stealing, even if you never process a single photon from a distant star.
The Separation Problem Nobody Talks About Enough
