Abstract

Neural rendering has demonstrated remarkable success in high-quality 3Dneural reconstruction and novel view synthesis with dense input views andaccurate poses. However, applying it to extremely sparse, unposed views inunbounded 360{\deg} scenes remains a challenging problem. In this paper, wepropose a novel neural rendering framework to accomplish the unposed andextremely sparse-view 3D reconstruction in unbounded 360{\deg} scenes. Toresolve the spatial ambiguity inherent in unbounded scenes with sparse inputviews, we propose a layered Gaussian-based representation to effectively modelthe scene with distinct spatial layers. By employing a dense stereoreconstruction model to recover coarse geometry, we introduce a layer-specificbootstrap optimization to refine the noise and fill occluded regions in thereconstruction. Furthermore, we propose an iterative fusion of reconstructionand generation alongside an uncertainty-aware training approach to facilitatemutual conditioning and enhancement between these two processes. Comprehensiveexperiments show that our approach outperforms existing state-of-the-artmethods in terms of rendering quality and surface reconstruction accuracy.Project page: https://zju3dv.github.io/free360/