Page - 163 - in Joint Austrian Computer Vision and Robotics Workshop 2020

segmentations. As this information was not present we chose to disambiguate the two classes by setting a Dice coefficient threshold. To figure out a suffi- ciently high Dice threshold we refer to the score-vs- dice scatter plot in the middle column of figure 4. Herethebluemargincorrespondstotheground-truth region, proposed by the detector. The orange clus- ter within this margin then corresponds to true neg- atives (correct segmentations), and suggests the dice thresholdof0.87. Figure 4 shows three plots for each of the five methods. In the following its columns are described indetail. Left column: inaddition to feature scatterplot the decisionboundaryandthescoringfunctionof there- spective detector are shown. In the following texts the orange test points falling outside the blue region will be referred to as the positives, points inside the blue region as thenegatives. MiddlecolumnshowsscatterplotsofDicevsout- lier scores. The horizontal line is the Dice thresh- old. The vertical lines are the thresholds of the scor- ing function proposed by the respective algorithms. ThefourquadrantscorrespondtoTNs,FPs,FNs,and TPs, respectively. These four numbers are typeset in the top center of the plot and the recalls and preci- sionscomputed thereofaredisplayed in the titles. Right column shows the ROC and Precision- Recall curves corresponding to the possible thresh- olds in the scoring function. The areas under these curves are abbreviated by auROC and auPR, respec- tively, andare displayed in the title. The performance numbers are summarized in Ta- ble1. In termsofprecision, theareasunderROCand PR,thekNNseemstobethemethodofchoice. How- ever, the OCSVM wins in term of recall, because of its steep narrow margin which determines the outlier score. While LOF and FB are of lower recall, they are less over-fitted than earlier two, and we can ob- serve an improvement when an ensemble of LOFs is aggregated into the FB. The MCD is easily inter- pretablebutunfortunatelynotperformingwell. Looking at the result of the well-fitting OCSVM, there are four FNs with a low Dice coefficient. In- vestigation on these revealed that such cases indeed might appear, because the area curves of the ground truth do not show a minimum around the middle of the slices, but have a nearly a rising shape. While the segmentation algorithm did not perform well on these scans, it still exhibits a convex fit to the area method Rec. Prec. auROC auPR FB 0.73 0.95 0.96 0.93 KNN 0.77 1.00 0.97 0.94 LOF 0.65 0.89 0.96 0.91 MCD 0.54 0.88 0.95 0.87 OCSVM 0.85 0.92 0.88 0.89 Table1: Summaryof results curve. Analyzing the twofalsepositives,oneof themap- pearedclosetotheOCSVMboundary. Thelessover- fitted detectors (e.g the MCD), however, have classi- fied this point correctly. The second false positive was a FP in all methods, except for the KNN. This could be because the ground truth data again shows an unusual shape: in contrast to the other ground truthshapes it startswithahighmaximum, thenfalls down,butdoesnot riseupagain. Therearefewaddi- tional ground truth curves having this kind of shape which we consider unusual. When the segmentation algorithm yields such a shape, it is more likely to be awrongsegmentation. WhetheranROCcurveshouldbeusedtoassessan outlier detector depends on the imbalance of the test set. Inthecurrentsetting, thesegmentationalgorithm [5] does not seem to be mature enough as it pro- duces around 25 percent of incorrect segmentation. As more reliable segmentation methods will be de- veloped, the test set becomes increasingly more im- balancedand thevalidationbyROCand its areawill have tobe replacedby theprecision-recall curves. 5.ConclusionandFutureWork We proposed a semi-supervised method to detect incorrectly segmented OCT retina scans: ground- truth segmentations are used, after feature extraction and projection to 2D, to train the decision bound- ary and the outlier scoring function. This function is subsequently used to flag the incorrectly segmented scans. We evaluated a selection of five outlier detection methods and find the results to be a promising start- ingpoint toaddress thegivenproblem. While in this work the data-pipeline components are tailored to a specific segmentation algorithm and itspitfalls,wewouldliketosketchhowthepresented approach can be generalized. Firstly, higher-degree polynomials(i.e.,moreregressioncoefficients)could be used if it turns out that the segmentations can not 163