What Changed
Traditional benchmarks for evaluating Large Language Models (LLMs) in scientific data analysis have primarily focused on code execution or workflow completion. This approach often fails to account for the distinct types of scientific claims that analysis supports, such as hypothesis exploration, statistical inference, and mechanistic explanation, each with unique assumptions and validity criteria. A new benchmark, SDABench, has been introduced to address this gap by reorganizing LLM evaluation around six specific scientific capabilities: descriptive, exploratory, inferential, predictive, causal, and mechanistic.
SDABench spans five scientific domains: Biology, Chemistry, Environment, Geography, and Physics. It comprises 527 real-data instances (SDA-Real) and 6000 synthetic instances (SDA-Synth), presented in both multiple-choice and open-ended formats. This comprehensive structure, constructed through an automated pipeline, aims to provide a more nuanced and scientifically relevant assessment of LLM capabilities.
Technical Details
SDABench's core innovation lies in its capability-oriented evaluation framework. Instead of merely checking if an LLM can execute a piece of code or complete a predefined workflow, the benchmark assesses how well an LLM can support different types of scientific claims. The six capabilities are defined as follows:
