Various combinations of cameras enrich computational photography, among which reference-based superresolution (RefSR) plays a critical role in multiscale imaging systems. However, existing RefSR approaches fail to accomplish high-fidelity super-resolution under a large resolution gap, e.g., 8× upscaling, due to the lower consideration of the underlying scene structure. In this paper, we aim to solve the RefSR problem in actual multiscale camera systems inspired by multiplane image (MPI) representation. Specifically, we propose Cross-MPI, an end-to-end RefSR network composed of a novel plane-aware attention-based MPI mechanism, a multiscale guided upsampling module as well as a super-resolution (SR) synthesis and fusion module. Instead of using a direct and exhaustive matching between the cross-scale stereo, the proposed plane-aware attention mechanism fully utilizes the concealed scene structure for efficient attention-based correspondence searching. Further combined with a gentle coarse-to-fine guided upsampling strategy, the proposed Cross-MPI can achieve a robust and accurate detail transmission. Experimental results on both digitally synthesized and optical zoom cross-scale data show that the Cross-MPI framework can achieve superior performance against the existing RefSR methods and is a real fit for actual multiscale camera systems even with large-scale differences.

[arXiv] [Code]

Fig 1. Method overview. We first estimate the initial alpha maps through a plane-aware attention-based MPI module. A novel multiscale guided upsampling module is then designed for generating super-resolved alpha maps. To transfer HR details of the reference view and generate the final SR result, the pipeline ends with an SR synthesis and fusion module. The whole pipeline is elaborately designed for the cross-scale stereo RefSR problem by fully considering camera relationships as well as the underlying scene structure.

Fig 2. Visual comparisons of different SR algorithms (8×) on RealEstate10K dataset.

Fig 3. Visual comparisons of different SR algorithms (8×) on RealEstate10K dataset. It is interesting to point out that only our method can provide SR result that maintains the original wall decorations.

Fig 4. (Left) Our datasets are composed of digital synthesized cross-scale data generated from RealEstate10K [12] (a), and optical zoomed cross-scale data (b) and (c). (Right) Test results on real cross-scale stereo pairs.