Page - 71 - in Joint Austrian Computer Vision and Robotics Workshop 2020

HPS:HolisticEnd-to-EndPanopticSegmentation NetworkwithInterrelations Gu¨ntherKniewasser,AlexanderGrabner,PeterM.Roth InstituteofComputerGraphicsandVision,GrazUniversityofTechnology,Austria {guenther.kniewasser@student,alexander.grabner@icg, pmroth@icg}.tugraz.at Abstract. To provide a complete 2D scene segmen- tation, panoptic segmentation unifies the tasks of se- mantic and instance segmentation. For this purpose, existing approaches independently address semantic and instance segmentation and merge their outputs in a heuristic fashion. However, this simple fusion has two limitations in practice. First, the system is notoptimized for thefinalobjective inanend-to-end manner. Second, themutual informationbetween the semanticand instancesegmentation tasks isnot fully exploited. To overcome these limitations, we present a novel end-to-end trainable architecture that gen- erates a full pixel-wise image labeling with resolved instance information. Additionally, we introduce in- terrelations between the two subtasks by providing instancesegmentationpredictionsas feature input to our semantic segmentation branch. This inter-task link eases the semantic segmentation task and in- creases the overall panoptic performance by provid- ing segmentation priors. We evaluate our method on the challenging Cityscapes dataset and show signif- icant improvements compared to previous panoptic segmentationarchitectures. 1. Introduction Panopticsegmentation[12]addresses theproblem of complete 2D scene segmentation by not only as- signing a class label to each pixel of an image but also differentiating between instances within a com- mon class. Thus, it can be seen as a unification of semantic segmentation [22, 24, 3] and instance seg- mentation [8, 13, 20, 16]. Panoptic segmentation is a new and active research area with applications in augmented reality, robotics, and medical imag- ing [5,23,30]. To predict a panoptic segmentation of an image, recent approaches perform three tasks. First, they Figure 1: Illustration of our proposed panoptic seg- mentation network with task interrelations. We pro- vide instance segmentation predictions as additional feature input to our semantic segmentation branch. In this way, we exploit a segmentation prior which increases theoverall panopticperformance. performsemanticsegmentationto identifyregionsof uncountable stuff classes like sky. Second, they per- form instance segmentation to detect individual in- stances of countable things classes like cars. Third, they merge the outputs of these two tasks into a sin- glepanopticprediction. However, this strategyhas twolimitations inprac- tice. First, because the panoptic output is generated using heuristics, the system cannot be optimized for the final objective in an end-to-end manner. Second, semanticand instancesegmentationsharemutual in- formation and similarities but the relation between the two tasks is not exploited because they are ad- dressed independently. Toovercometheselimitations,weproposeaholis- tic end-to-end trainable network for panoptic seg- mentation (HPS) with interrelations between the se- mantic and the instance segmentation branches, as shown in Figure 1. Our network directly generates a full pixel-wise image labeling with resolved in- stance informationbyusingdifferentiableoperations instead of heuristics to combine individual results. Moreover, to take advantage of mutual information between the semantic and the instance segmentation 71