JetBrains Research
Research is crucial for progress and innovation, which is why at JetBrains we are passionate about both scientific and market research
Research
In any recent model comparison, you’ve probably seen a single number from a coding benchmark to represent “How good this model is at coding”. Those scores are tempting: they’re simple, they’re leaderboard-friendly, and they seem to tell a clear story about progress.
But that story didn’t hold up when we looked more closely at benchmarks that are considered standards. The models dramatically improved on the exact tasks we trained them on, yet those gains often did not show up on other benchmarks or on slightly different tasks in the same codebase. This blog post is about our research on that meaning gap between what coding benchmarks measure and what we wish they measured – and how to improve these benchmarks. This paper will be presented by our team at the Deep Learning for Code (DL4C) workshop, co-located with the International Conference on Machine Learning (ICML) this July in Seoul.
