The scariest moment I've had while talking design with an AI wasn't a code bug. It wasn't a prompt gone sideways. It was this: the AI (and honestly, me too) had quietly decided "this data is fine to use" without asking a single soul.
What I was doing
I was hashing out the design for a business I run on the side, with Claude as my sparring partner. The specific wall I was throwing balls at: how to structure the constraints a certain feature has to protect. I stood up proxies — subagents playing the roles of on-the-ground staff, the exec, the architect — to split up the arguments and pour it all into a design memo. So far, so smooth.
Partway through, I remembered something. Field records from a separate contract gig I take on. They hold pretty sensitive personal information — staff health, family situations, near-miss incident reports. First-class material for pressure-testing where the design actually bites. "Reference this too," I told the AI.
And the AI looked like it behaved. It copied over zero concrete values, zero names, zero numbers, and generalized everything up to the category level into a validation section of the memo. It even said it out loud: "No PII baked in." And then it wrote this:
