Upstream will add a field, rename a field, change a type from string to object, and start sending null where it never did. None of that is avoidable. What is avoidable is finding out about it from a crash in a consumer at three in the morning.
Three schemas, not one
A document pipeline has three separate schemas and they evolve independently. Conflating them is the root of most of the pain.
The source schema — whatever the upstream system emits. You do not control it and you frequently are not told when it changes.
The internal schema — your normalised document and chunk representation. You control this completely; it should change rarely and deliberately.
