Abstract

This paper targets the challenge of real-time LiDAR re-simulation in dynamicdriving scenarios. Recent approaches utilize neural radiance fields combinedwith the physical modeling of LiDAR sensors to achieve high-fidelityre-simulation results. Unfortunately, these methods face limitations due tohigh computational demands in large-scale scenes and cannot perform real-timeLiDAR rendering. To overcome these constraints, we propose LiDAR-RT, a novelframework that supports real-time, physically accurate LiDAR re-simulation fordriving scenes. Our primary contribution is the development of an efficient andeffective rendering pipeline, which integrates Gaussian primitives andhardware-accelerated ray tracing technology. Specifically, we model thephysical properties of LiDAR sensors using Gaussian primitives with learnableparameters and incorporate scene graphs to handle scene dynamics. Building uponthis scene representation, our framework first constructs a bounding volumehierarchy (BVH), then casts rays for each pixel and generates novel LiDAR viewsthrough a differentiable rendering algorithm. Importantly, our frameworksupports realistic rendering with flexible scene editing operations and varioussensor configurations. Extensive experiments across multiple public benchmarksdemonstrate that our method outperforms state-of-the-art methods in terms ofrendering quality and efficiency. Our project page is athttps://zju3dv.github.io/lidar-rt.