Robotics foundation models have made remarkable progress. Today’s best systems can follow natural language instructions to pick, place, sort, and manipulate a wide variety of objects. But as these models grow more capable, evaluating them rigorously has become one of the field’s hardest unsolved problems. In this blog post, we introduce the key problems and our method for addressing them.
Why current benchmarks fall short
Real-world testing is expensive, slow, and difficult to reproduce. For a robot’s performance in the real world to be evaluated thoroughly, we need a reasonable proxy. Simulation is the natural place to run large-scale robot evaluations. Yet most existing benchmarks share a few critical issues.
Visual domain overlap in training and evaluation
First, the data and environments used in policy training and evaluation are almost always drawn from the same visual source. When a model is fine-tuned on simulated data and evaluated in that same simulated environment, strong performance reveals only that the model memorized the setup, not that it can generalize. This remains a critical issue in robot evaluations, as the visual quality of simulation hasn’t achieved parity with real-world image observations. Real2sim approaches address this issue by reconstructing photorealistic environments from real-world images using techniques like Gaussian Splatting, but per-scene setup can exceed an hour, making large-scale testing impractical.
