GitGuardian helps developers and security teams detect secrets (API keys, tokens, credentials) that have been accidentally committed to source code. At the core of our platform sits our secret detection engine: a component that takes raw bytes as input and outputs detected secrets, running against hundreds of gigabytes of code and data every day. Migrating this engine to Rust was driven by two goals: raw performance gains to reduce infrastructure costs, and improved portability, allowing us to eventually run the engine in environments where a Python runtime is not available.
Rewriting legacy software in Rust has been a trend in recent years. So much that it did not only spawn numerous rewrites but also its own meme. When we decided to migrate our secret detection engine to Rust, one of our main goals was performance: Our Python implementation was fast, but given that we scan hundreds of gigabytes on a daily basis, even small improvements can massively reduce our costs.
We set out on this journey with an audacious goal: Rewrite the engine from scratch and have nobody notice it. In retrospect, the decision to avoid breaking changes as much as possible ended up causing most of our headaches during the migration. The plan was to change the plane's engine, mid-flight, with all business class passengers on board.
