For a paper published in the International Journal of Applied Cryptography, a team compared 11 leading large language models (LLMs) for software security. They found that no single system consistently outperforms its rivals in detecting vulnerabilities. This, they suggest, means organizations must select such tools according to the specific software they are analyzing.
The study assessed open-source and proprietary LLMs across four public benchmark data sets covering Android applications, Internet of Things (IoT) software and blockchain smart contracts. The researchers also tested whether the models could identify privacy-invasive behavior in code and whether retrieval-augmented generation (RAG), a technique that supplements an AI model with external information during use, could improve detection.
The findings come as software vulnerabilities continue to rise. Industry reports cited by the authors suggest an almost two-thirds annual increase in newly discovered vulnerabilities compared with the previous year. Vulnerabilities that have been exploited have increased by 96%. They add that software supply chain attacks, which target the software development and distribution process, have also increased sharply.
