A new model from Google Deepmind called GenCeption uses a pre-trained video generation model as the basis for classic computer vision tasks. It achieves state-of-the-art performance in depth estimation, segmentation, and 3D pose estimation while needing very little training data.
Language models became versatile processing systems almost as a byproduct of learning to predict the next word. That task appears to require models to absorb grammar, world knowledge, and contextual relationships during training. This is the prevailing explanation for the emergent capabilities of large language models.
Computer vision still lacks an equivalent training method. Specialized models dominate the field, including "Segment Anything" for segmentation and "Depth Anything" for depth estimation. Each uses its own architecture.
GenCeption generates depth maps, surface normals, segmentation masks, and pose estimates from the same video based on a text prompt. Despite training mostly on synthetic data, it works on real footage and object classes it never encountered during training. | Image: Google Deepmind
In a new paper, Deepmind researchers argue that large text-to-video models could bridge this gap. Generating realistic videos requires understanding the spatial geometry of a scene, how objects move, and basic physics. These models also use text descriptions during training, which links language to visual content. They can train on vast datasets because labeling the data is relatively cheap.
