Deduplication and decontamination are the same computation pointed at two different questions: is this row a copy of another row in the same set, and is this training row a copy of something in the set I am going to be judged on?
Two jobs that share one technique
Deduplication removes near-identical rows within a corpus. The costs of not doing it are concrete: duplicated examples get extra weight in training for no reason anyone chose, they inflate every count you report, they waste tokens, and they raise memorisation of the duplicated text.
Decontamination removes training rows that overlap your evaluation data. Skipping it produces a score that is partly a measurement of memorisation, and the direction of the error is always flattering — which is why it is so rarely noticed by the person who benefits.
Both reduce to the same underlying question — is document A substantially contained in document B — so one implementation serves both, with different reference sets on the other side of the comparison.
