Feed-forward 3D Gaussian Splatting (3DGS) models enable real-time scene generation but are hindered by suboptimal pixel-aligned primitive placement, which relies on a dense, rigid grid and limits both quality and efficiency. We introduce a new feed-forward architecture that detects 3D Gaussian primitives at a sub-pixel level, replacing the pixel grid with an adaptive, "Off The Grid" distribution. Inspired by keypoint detection, our multi-resolution decoder learns to distribute primitives across image patches. This module is trained end-to-end with a 3D reconstruction backbone using self-supervised learning. Our resulting pose-free model generates photorealistic scenes in seconds, achieving state-of-the-art novel view synthesis for feed-forward models. It outperforms competitors while using far fewer primitives, demonstrating a more accurate and efficient allocation that captures fine details and reduces artifacts. Moreover, we observe that by learning to render 3D Gaussians, our 3D reconstruction backbone improves camera pose estimation, suggesting opportunities to train these foundational models without labels.
We fine-tune VGGT to predict and render 3D Gaussians in a self-supervised loop that does not require 3D annotations for training.
Our 3D Gaussians Decoder learns to detect and describe 3D primitives from 2D images, depths and latent features.
Fine-tuning VGGT for rendering with our method enables to avoid depth estimation failure cases in specular or sky areas.
We also observe that our fine-tuning improves camera pose and focal length estimations, check the paper for more details !
@article{moreau2023human,
title={Off The Grid: Detection of Primitives for Feed-Forward 3D Gaussian Splatting},
author={Moreau, Arthur and Shaw, Richard and Nazarczuk, Michal and Shin, Jisu and Tanay, Thomas and Zhang, Zhensong, and Xu, Songcen and P{\'e}rez-Pellitero, Eduardo},
journal={arXiv preprint},
year={2025}
}