Abstract

Reconstructing complex reflections in real-world scenes from 2D images isessential for achieving photorealistic novel view synthesis. Existing methodsthat utilize environment maps to model reflections from distant lighting oftenstruggle with high-frequency reflection details and fail to account fornear-field reflections. In this work, we introduce EnvGS, a novel approach thatemploys a set of Gaussian primitives as an explicit 3D representation forcapturing reflections of environments. These environment Gaussian primitivesare incorporated with base Gaussian primitives to model the appearance of thewhole scene. To efficiently render these environment Gaussian primitives, wedeveloped a ray-tracing-based renderer that leverages the GPU's RT core forfast rendering. This allows us to jointly optimize our model for high-qualityreconstruction while maintaining real-time rendering speeds. Results frommultiple real-world and synthetic datasets demonstrate that our method producessignificantly more detailed reflections, achieving the best rendering qualityin real-time novel view synthesis.