Page - 160 - in Proceedings of the OAGM&ARW Joint Workshop - Vision, Automation and Robotics

Depth-guided Disocclusion Inpainting for Novel View Synthesis* Thomas Rittler1,2, Matej Nezveda1,2, Florian Seitner2, and Margrit Gelautz1 Abstract—The generation of novel views is a crucial process- ing step in 3D content generation, since it gives control over the amount of depth impression on (auto-)stereoscopic devices and enables free-viewpoint video viewing. A critical problem in novel view generation is the occurrence of disocclusions caused by a change in the viewing direction. Thus, areas in the novel views may become visible that were either covered by foreground objects or were located outside the borders in the original views. In this paper, we propose a depth-guided inpainting approach which relies on efficient patch matching to complete disocclusions along foreground objects and close to the imageborders.Ourmethodadapts itspatchsizesdepending onthedisocclusionsizesand incorporates thedepth information byfocusingonthebackgroundscenecontent forpatchselection. A subjective evaluation based on a user study demonstrates the effectiveness of the proposed approach in terms of quality of the 3D viewing experience. I. INTRODUCTION The generation of novel views from an existing single view and its corresponding depth map is a crucial processing step for 3D content generation and processing. Such newly generated views enable the users to watch 3D content on different types of 3D displays, including multi-user au- tostereoscopic devices with a comfortable range of viewing perspectives, and navigate in 3D space for free-viewpoint video applications. The 2D input image and its associated depth map – known as 2D-plus-depth [11] – can be delivered by a variety of sources such as depth sensors based on time- of-flight or structured light (e.g., Microsoft’s Kinect), stereo cameras, or 2D-to-3D conversion techniques. A principal problem in novel view generation is the occurrence of disocclusions due to a change in the viewing direction. Some areas in the original views that were either covered by a foreground object or were located outside the image borders may become visible in the novel views. To deal with these disocclusions, one common approach is to pre-process the depth maps. In particular, filtering techniques are applied to the associated depth maps prior to the novel view generation [16]. Although this approach can reduce the appearance of disocclusions, it can also lead to spatial distortions in the scene geometry of the novel views. Another approach is to use image inpainting techniques to fill in the disoccluded areas in the novel views with *This work was supported by the Technology Agency of the City of Vienna (ZIT) under the project PAINT3D and finalized under the project Precise3D, funded by the Austrian Research Promotion Agency (FFG) and the Austrian Ministry BMVIT under the program ICT of the Future. 2 emotion3D GmbH, Gartengasse 21/3, 1050 Vienna, Austria; {nezveda, seitner}@emotion3d.tv 1 Institute of Software Technology and Interactive Systems, Vienna Uni- versity of Technology, Favoritenstrasse 9-11/188-2, 1040 Vienna, Austria; margrit.gelautz@tuwien.ac.at suitable estimates derived from the visible scene content. However, traditional inpainting algorithms (e.g., [5]) do not take into account additional knowledge provided by the depth data. For that reason, several inpainting strategies have been proposed that incorporate depth information during disocclusion filling [6], [8], [10], [13], [1], [15], [14]. While most related work aims at rendering photorealistic views, suitable inpainting approaches may also be required in the context of non-photorealistic rendering [9]. A few depth- induced inpainting strategies build upon PatchMatch (PM) [2], which is a randomized search algorithm that quickly finds correspondences between disjoint image patches. For example, He et al. [10] add the depth information to the PM algorithm by restricting the validity of patches used for inpainting. However, as their method was initially proposed for foreground object removal, the authors rely on a-priori depth information in the region to be filled which is not available when considering disocclusions. Morse et al. [13] extend PM from single image completion to stereo image pairs by not only incorporating depth information extracted from the stereo pairs but also allowing the matching of patches across the stereo pairs. However, the additional orig- inal view of a stereo image pair is not available in a 2D-plus- depth setup as considered in this work. Additionally, none of the aforementioned depth-guided inpainting approaches considers subjective quality assessment in the evaluation of their results. However, the results of Bosc et al. [3] indicate the need of subjective quality assessment in terms of novel views evaluation, as commonly used 2D quality metrics do not reflect the subjective quality of novel views. A very recent publication [4] gives an in-depth evaluation using the Middlebury ground truth data set, but does not incorporate user studies. In this paper, we propose a depth-guided inpainting ap- proach for disocclusion filling in novel views based on PM. Our approach incorporates the supplementary depth information to favor background patches during the disoc- clusion inpainting and uses adaptive patch sizes for efficient hole filling. We perform a paired comparison user study to evaluate our inpainting results in the context of stereoscopic viewing and present experimental results that show that our depth-guided inpainting approach yields better subjective quality compared to several earlier approaches. The rest of the paper is organized as follows: Section 2 describes theproposed inpaintingmethod.Section3provides details on our experimental setup. Section 4 presents the re- sults of the user study along with some inpainting examples, and Section 5 concludes the paper. 160