A refined weighted mode filtering approach for depth video enhancement

Given a low-quality depth video and its corresponding high-quality color video, we intend to improve depth quality by suppressing both spatial and temporal noise. A refined weighted mode filtering method (WMF) based on a joint histogram is proposed. For WMF, similarity between reference and neighbor pixels plays an important role in counting each bin of the joint histogram. Since calculating similarity using single pixel will be affected by random pixel noise, we utilize patch-based NL-means (Non-Local means) for structure-aware similarity calculation; also, we fuse color and depth similarity adaptively with credibility maps to deal with texture copying problem. For temporally consistent recovery, we introduce inter-frame correlation by integrating neighboring frames with optical flow and patch-based similarity measurement. Experimental results show that our proposed method has achieved more complete and clear depth, especially in discontinuous areas. Furthermore, temporally enhancement of depth video addresses flickering problem and gets more stable depth.