LMG: Efficient Streaming of Layered Mesh–Gaussian 3D Scenes

Delivering immersive 6-DoF experiences over capacity-constrained networks requires balancing visual fidelity, interactive capabilities, and streaming efficiency. Existing polygonal mesh or 3D Gaussian Splatting (3DGS) approaches exhibit fundamental trade-offs among these requirements, whereas prior hybrid methods have largely focused on reconstruction quality rather than end-to-end delivery.

In this work, we introduce LMG, the first end-to-end streaming system utilizing a Layered Mesh-Gaussian representation for the efficient and interactive delivery of static 3D scenes. Our model organizes textured meshes into a streamable base layer to ensure structural interactivity and attaches surface-aligned Gaussians into an enhancement layer for high-fidelity, photorealistic rendering. To enable effective deployment, we develop a complete system comprising a production server, a streaming server, and an XR client. The production server features a joint training pipeline with a custom hybrid rasterizer and budget-aware allocation algorithms that concentrate Gaussians in perceptually important regions. The streaming server employs a layered and tile-based pipeline tailored for LMG, prioritizing the delivery of the base layer for a quick scene overview, followed by the progressive refinement of visible regions using Gaussians.

Evaluations of our systems demonstrate that the LMG production server significantly outperforms the state-of-the-art baseline in visual quality--improving PSNR by up to 10.99 dB (8.89 dB on average) and SSIM by up to 0.293 (0.255 on average). Furthermore, our results highlight the efficacy of the LMG streaming system, showing that layered and tiled streaming yields quality improvements of up to 19.81 dB in PSNR (5.80 dB on average), 0.143 in SSIM (0.06 on average), and 40.86 in VMAF (16.06 on average), while enabling low-latency and interactive XR experiences.