Page - 161 - in Proceedings of the OAGM&ARW Joint Workshop - Vision, Automation and Robotics

II. PROPOSED APPROACH Wesuggestan inpainting technique thatbuildsupon PM as an efficient strategy for finding patch correspondences based on color differences. The proposed approach incorporates adaptive patch sizes and search space restrictions based on depth information, as explained in the following subsections. First, the formalism of the general inpainting problem is recapped [5]: Let I be an input image andΩ⊆ I a “hole” to be filled, called the target region. That is,Ω denotes all the missing pixels within I. Additionally, the source region Φ provides samples used in the infilling process. The goal is now to complete the missing regionΩwith data fromΦ so that the resulting image will be visually coherent. While conventionallyΦ= I\Ω, we restrictΦ to candidates from the image background as part of our approach. A. Adaptive patch sizes As opposed to iterative inpainting approaches that shrink the holes by successively copying patches of constant size, we perform the inpainting step only once at the end of the image completion chain, with the goal to avoid propagating erroneous inpainting results from one iteration step to the next. Our non-iterative approach is enabled by the usage of adaptive patch sizes. If fixed-size patches are used and the patch size is smaller than the size ofΩ, there are some target patches containing no valid image information (see blue rectangle in Fig. 1a) that is required to compute the patch similarities. For that purpose, a threshold τ1 is specified to ensure a minimum percentage of valid pixels in each target patch. The corresponding patch size for each target pixel is determined by successively incrementing the patch dimensions until the percentage of the valid source pixels exceeds τ1. Hence, the selected patches are smaller near the borders and are growing as the patch’s central pixel is moving towards the hole’s centroid, as illustrated in Fig. 1b. As a side effect, fewer patches are involved in the color synthesis of an individual pixel (based on weighted color averaging of overlapping patches)near theboundariesofΩ,whichhelpsavoidblurring artifacts in these regions. By introducing adaptive patch sizes it is guaranteed that themajorityof the targetpatchescontainacertainpercentage of valid pixels. However, there may arise situations where the combination of target and source patches becomes im- practical, as schematically illustrated in Fig. 1c. Hence, a second threshold τ2 (equal to or smaller than τ1) is specified to maintain the majority of valid pixels in the matching step and to ensure a minimal overlap between valid pixels of the target patch and the corresponding source patch. B. Depth There are two major reasons for disocclusions that cause blankareas innovelviews: (a)areas thathadbeencoveredby a foreground object in the original view, and (b) areas along the image borders that had been outside the field of view in the original image. While scene depth is not taken into account when dealing with case (b), it is reasonable to fill Fig. 1. Schematical overview of the basic concepts of our inpainting approach: (a) constant versus (b) adaptive patch size; (c) problem of non- overlapping valid pixels between target and source patch; (d) target patch comprising foreground and background pixels. Further details are given in the text. occlusions of group (a) with image data obtained from back- ground regions. As these holes emerge due to sharp depth transitions (i.e., depth discontinuities) at object boundaries, a target patch may comprise pixels that belong to foreground objects as well as pixels that are part of the background, as illustrated in Fig. 1d. Consequently, inpainting artifacts occur – hereinafter also referred to as foreground color blur – which are caused by color bleeding from the foreground. Therefore, depth information is incorporated in the matching stage to find appropriate patch correspondences and prevent foreground regions from being used for filling disoccluded regions. Since depth information is not available in the target region, the depth values have to be synthesized first from the warped depth values in the surrounding. For every hole inΩ, each scanline is first filled by a constant value determined as the maximum depth value of the left and right pixel located at the hole boundary. Then, the minimum of the newly filled in depth values is selected as a lower bound of permissible depth levels in the nearest-neighbor search for target patches of the respective hole. An additional outlier removal based on the statistics of the depth histogram is applied to make the procedure more robust to depth map inaccuracies. III. EXPERIMENTAL SETUP In order to investigate the effectiveness of our proposed inpainting algorithm on the perceived quality of stereoscopic images, a pair-wise comparison study was conducted. The stereo pairs used for evaluation were formed by the original left views and novel right views, i.e., synthesized views derived from the left views and the corresponding depth maps with disocclusions filled by inpainting. This section 161