Dense-view synthesis for three-dimensional light-field display based on unsupervised learning

Three-dimensional (3D) light field display, as a potential future display method, has attracted considerable attentions. However, there still exist certain issues to be addressed, especially the capture of dense views in real 3D scenes. Using sparse cameras associated with view synthesis algorithm has become a practical method. Supervised convolutional neural network (CNN) is used to synthesize virtual views. However, such large amount of training target views is sometimes difficult to be obtained and the training position is relatively fixed. Novel views can also be synthesized with unsupervised CNN, but the method is difficult to handle the case of free-posed views as long as non-uniform full parallax exists. Here, a method of dense-view synthesis based on unsupervised learning is presented, which can synthesize arbitrary virtual views with multiple free-posed views captured in the real 3D scene based on unsupervised learning. Multiple posed views are reprojected to the target position and input into the neural network. The network outputs a color tower and a selection tower indicting the scene distribution along the depth direction. A single image is yielded by the weighted summation of two towers. The proposed network is end-to-end trained based on unsupervised learning by minimizing errors during reconstructions of posed views. A virtual view can be predicted in a high quality by reprojecting posed views to the desired position. And a sequence of dense virtual views can be generated for 3D light-field display by repeated predictions. Experimental results demonstrate the validation of our proposed network. PSNR of synthesized views are around 30dB and SSIM is over 0.90. We believe the proposed approach will contribute to the wide applications of 3D light-field display in the future.