A single viral clip never stays a single file. Within hours of a video trending, our crawlers pull the "same" clip back a dozen times: re-encoded at a lower bitrate, letterboxed for a vertical feed, watermarked by an aggregator, trimmed by three seconds, or mirrored with a 2% zoom to dodge platform fingerprinting. A byte-for-byte SHA-256 sees twelve unique videos. Our users see one clip spammed across their feed. For a European viral-video discovery product where feed quality is the value proposition, that is an existential bug, not a cosmetic one. At ViralVidVault we lean on perceptual hashing to collapse those near-duplicates before anything reaches the ranking model.
This post is the actual approach we run in production: how to turn frames into hashes that survive re-encoding, how to aggregate per-frame hashes into a per-video fingerprint, and how to query millions of them without a GPU or a dedicated vector database. The heavy lifting is Python, the storage and query layer is PHP 8.4 on SQLite in WAL mode, and the hot comparison path is a small Go helper. No proprietary services, nothing that ships a raw frame off our origin.
Why cryptographic hashing is the wrong tool
Cryptographic hashes are designed to be maximally sensitive: flip one bit of input and roughly half the output bits change. That is exactly what you want for integrity checking and exactly what you do not want for deduplication. Two visually identical frames that differ only because one went through an extra H.264 pass will produce completely unrelated SHA-256 digests. There is no notion of "close."
