Real-time Illumination Estimation for Mixed Reality on Mobile Devices

We present a lightweight lighting estimation method for the purpose of realistic mixed reality (MR) on mobile devices. Given a single RGB image, our method estimates the environment lighting and renders the virtual object in real-time. While previous works tackled this problem by reconstructing the complicated high dynamic range (HDR) environment maps, our deep neural network directly infers the corresponding spherical harmonics in extensive environments. Compared to previous approaches, our method is more robust and efficient as it works for both indoor and outdoor scenes in real-time. Experiments show that our approach achieves realistic rendering in various MR scenarios.