What Changed
In the rapidly evolving field of Vision-Language-Action (VLA) models, a fundamental architectural challenge has persisted: the frame mismatch between visual observation and motor action. Standard VLA models typically ingest visual data from the perspective of a camera, yet they are tasked with producing actions defined in the robot's own 3D coordinate frame. In controlled, laboratory environments where the camera is fixed, this mismatch is often masked; the model effectively memorizes the specific mapping from the camera's viewpoint to the robot's required movement. However, as the industry shifts toward large-scale, diverse datasets that aggregate demonstrations across varied camera placements and robot configurations, this memorization strategy fails. The model struggles to generalize because the relationship between the camera-frame visual input and the robot-frame action changes whenever the camera moves. The introduction of robot-centric pointmaps represents a significant shift in how these models perceive their environment, moving away from raw RGB camera frames toward a representation that is inherently aligned with the robot's own physical coordinate system.
Technical Details
