MirrorGaussian: Reflecting 3D Gaussians for Reconstructing Mirror Reflections

Jiayue Liu1*   Xiao Tang2*   Freeman Cheng3   Roy Yang2   Zhihao Li2†   Jianzhuang Liu4  
Yi Huang4   Jiaqi Lin1   Shiyong Liu2   Xiaofei Wu2   Songcen Xu2   Chun Yuan1†
1Tsinghua University     2Huawei Noah's Ark Lab     3University of Toronto     4Chinese Academy of Sciences
jy-liu22@mails.tsinghua.edu.cn   tangxiao12@huawei.com   zhihao.li@huawei.com   yuanc@sz.tsinghua.edu.cn
* Equal contribution     † Corresponding authors
ECCV 2024
arXiv

The first two video clips show the high-quality rendering results of our method on various datasets
while the third demonstrates the real-time rendering performance of our method.

Abstract

3D Gaussian Splatting showcases notable advancements in photo-realistic and real-time novel view synthesis. However, it faces challenges in modeling mirror reflections, which exhibit substantial appearance variations from different viewpoints. To tackle this problem, we present MirrorGaussian, the first method for mirror scene reconstruction with real-time rendering based on 3D Gaussian Splatting. The key insight is grounded on the mirror symmetry between the real-world space and the virtual mirror space. We introduce an intuitive dual-rendering strategy that enables differentiable rasterization of both the real-world 3D Gaussians and the mirrored counterpart obtained by reflecting the former about the mirror plane. All 3D Gaussians are jointly optimized with the mirror plane in an end-to-end framework. MirrorGaussian achieves high-quality and real-time rendering in scenes with mirrors, empowering scene editing like adding new mirrors and objects. Comprehensive experiments on multiple datasets demonstrate that our approach significantly outperforms existing methods, achieving state-of-the-art results.

Architecture

Overview of MirrorGaussian. MirrorGaussian is grounded on the mirror symmetry between the real-world scene and its counterpart in the mirror. We first reflect the 3D Gaussians $P_r$ about the mirror plane $\mathcal{P}$ to obtain its mirrored counterpart $P_m$. Then, we rasterize $P_r$ to get the real-world image $I_r$ and the mirror mask $M$, and rasterize $P_m$ to get the mirror image $I_m$. The final image $I$ is composited by $I_r$ and $I_m$ using $M$. $I$ and $M$ are supervised by the captured image $I_{gt}$ and its annotated mirror mask $M_{gt}$, respectively. Note that for the sake of visual simplicity, $P_r$ and $P_m$ have been cropped.

Comparison With SOTA

Architecture
Vanilla 3DGS MS-NeRF MirrorGaussian
Vanilla 3DGS MS-NeRF MirrorGaussian
Vanilla 3DGS MS-NeRF MirrorGaussian
Vanilla 3DGS MS-NeRF MirrorGaussian

Scene Editing

Please click the videos for better view.

BibTeX

@inproceedings{liu2024mirrorgaussian,
    title={MirrorGaussian: Reflecting 3D Gaussians for Reconstructing Mirror Reflections},
    author={Liu, Jiayue and Tang, Xiao and Cheng, Freeman and Yang, Roy and Li, Zhihao and Liu, Jianzhuang and Huang, Yi and Lin, Jiaqi and Liu, Shiyong and Wu, Xiaofei and others},
    booktitle={ECCV},
    year={2024}
  }