Abstract

Thermography is especially valuable for the military and other users ofsurveillance cameras. Some recent methods based on Neural Radiance Fields(NeRF) are proposed to reconstruct the thermal scenes in 3D from a set ofthermal and RGB images. However, unlike NeRF, 3D Gaussian splatting (3DGS)prevails due to its rapid training and real-time rendering. In this work, wepropose ThermalGaussian, the first thermal 3DGS approach capable of renderinghigh-quality images in RGB and thermal modalities. We first calibrate the RGBcamera and the thermal camera to ensure that both modalities are accuratelyaligned. Subsequently, we use the registered images to learn the multimodal 3DGaussians. To prevent the overfitting of any single modality, we introduceseveral multimodal regularization constraints. We also develop smoothingconstraints tailored to the physical characteristics of the thermal modality.Besides, we contribute a real-world dataset named RGBT-Scenes, captured by ahand-hold thermal-infrared camera, facilitating future research on thermalscene reconstruction. We conduct comprehensive experiments to show thatThermalGaussian achieves photorealistic rendering of thermal images andimproves the rendering quality of RGB images. With the proposed multimodalregularization constraints, we also reduced the model's storage cost by 90%.Our project page is at https://thermalgaussian.github.io/.