Page - 140 - in Proceedings of the OAGM&ARW Joint Workshop - Vision, Automation and Robotics

Generative Adversarial Network based Synthesis for Supervised Medical Image Segmentation* Thomas Neff1, Christian Payer1, Darko Sˇtern2, Martin Urschler2 Abstract—Modern deep learning methods achieve state-of- the-art results in many computer vision tasks. While these methods perform well when trained on large datasets, deep learning methods suffer from overfitting and lack of gener- alization given smaller datasets. Especially in medical image analysis, acquisition of both imaging data and corresponding ground-truth annotations (e.g. pixel-wise segmentation masks) as required for supervised tasks, is time consuming and costly, sinceexpertsareneededtomanuallyannotatedata. Inthiswork westudythisproblembyproposinganewvariantofGenerative AdversarialNetworks (GANs),which, inadditiontosynthesized medical images, also generates segmentation masks for the use in supervised medical image analysis applications. We evaluate our approach on a lung segmentation task involving thorax X-ray images, and show that GANs have the potential to be used for synthesizing training data in this specific application. I. INTRODUCTION Modern machine learning methods based on deep neural network architectures require large amounts of training data to achieve the best possible results. For standard com- puter vision problems, large datasets, such as MNIST [12], CIFAR10 [10], or ImageNet [23], containing millions of images, are publicly available. In the medical field, datasets are typically smaller by several orders of magnitude, as the acquisition process of medical images is costly and time consuming. Furthermore, ethical concerns make it harder to publicly release and share datasets. Finding methods to improve performance when training deep learning methods on small datasets is an area of active research. Recent work in the medical imaging domain has shown that it is possible to improve performance with small datasets by putting application specific prior knowledge into a deep neural network [17]. Another approach has been made popular by the U-Net [21] architecture for biomedical image segmentation, which demonstrated how strong data augmentation can be used to deal with low amounts of training data in deep network architectures. Even though data augmentation is simple to implement and achieves good results, it isonlyable toproducefixedvariationsofanygiven dataset, requiring the augmentation to fit the given dataset. Transfer learning approaches such as [19] show that training on large datasets (e.g. ImageNet) followed by fine- tuning on a small dataset achieves state-of-the-art results for *Thisworkwassupportedby theAustrianScienceFund(FWF):P28078- N33. 1Thomas Neff and Christian Payer are with the Institute for Com- puter Graphics and Vision, Graz University of Technology, Austria thomas.neff@student.tugraz.at 2Darko Sˇtern and Martin Urschler are with Ludwig Boltz- mann Institute for Clinical Forensic Imaging, Graz, Austria martin.urschler@cfi.lbg.ac.at datasets consisting of natural images. For medical imaging, the learned features from large natural image datasets may not be suitable, as the image features are very different com- pared to natural images. Furthermore, there is no straight- forward way of transferring 2D features to 3D features, which poses a limitation when working with 3D medical images. Due to the difference in features between medical and natural images, another approach is to use unsupervised feature extractors (e.g. Autoencoders [27]) which are trained on medical images only. Nevertheless, transferring weights learned by these unsupervised methods requires the target network architecture to be close to the source architecture, which is rarely the case. The requirement for large amounts of training data also popularized image generation methods in deep learning con- texts. Recently, research has shown that Generative Adver- sarial Networks (GANs) [4] can be used for a large variety of applications such as image-to-image translation [6] or unsu- pervised representation learning [18]. GANs have also been successfully used for unsupervised domain adaptation [8] of multi-modal medical imaging data, demonstrating their potential for use with small medical imaging datasets. Our goal was to use GANs in a completely different way, by using the high quality of the generated images to augment our small set of training data. We propose a novel modifica- tion to GANs, which generates new, synthetic images as well as thecorrespondingsegmentationmasks fromrandomnoise. This allows us to use the synthetic data as training data for a supervised segmentation task. We show that this architecture manages to produce convincing segmentation masks for the generated images. We evaluate the generated images in two different scenarios on an image segmentation task and show that training on purely generated images achieves results comparable to trainingonreal images forverysmalldatasets. II. RELATED WORK A. Training Data Augmentation Training data augmentation is a commonly used method to reduce the effects of overfitting with small training datasets as well as improve the generalization of the trained network. Most machine learning frameworks allow for simple aug- mentation such as rotation, translation or intensity shifts of training data. AlexNet [11] was one of the first convolutional neural network (CNN) architectures to implement online data augmentation with successful results. However, data augmentation only achieves good results if the augmentation can actually occur in the data, and is relevant to the required application. For medical imaging, elastic deformations [21] 140