Page - 117 - in Proceedings of the OAGM&ARW Joint Workshop - Vision, Automation and Robotics

Eight models are assigned to each parameter-typeα∈S. A model is reconstructed with the full range of parameter values, which are 8 values for ColourCorrection, PoseEs- timation, MaxIterations, and 5 values for SmoothnessTerm. The parameter space is discrete and the values form the individual parameter value setsPα. In a questionnaire 32 people (16 male, 16 female) between 19 and 55 years old, were asked to choose the most aesthetic reconstruction for each model. Since every reconstruction is mapped to a parameter value, they implicitly chose the parameter value which led to the best reconstruction quality. The best parameter choices according to the human votes havebeencounter-checked toproduce reasonable reconstruc- tions. During this result preparation, one model which was assigned to SmoothnessTerm, had to be omitted because of inconsistencies in the data. In detail the parameter value with the most human votes for this model leads to a failed reconstruction similar to the bottom right picture of Figure 4. Therefore the final dataset consists of 31 human judged model reconstructions.1 B. Evaluation Criteria Figure 2(a) and Figure 2(b) show example distributions of human decisions for specific models. One can see the variances in the votes. To cover these variances we define the following correctness criterion: Definition 1. Aparameter value estimation is correct if it is inside µ±σ of the human decisions. To test this criterion, human judgements have been simu- lated with random values. In detail for each decision distri- bution (e.g. Subfigure 2(b)) a uniformly distributed random value in the same discrete parameter range was generated. If the random value fulfilled the correctness criterion for the decision distribution, it was counted as correct, otherwise as incorrect. With 1000 trials this lead to a mean accuracy of 0.5095 and σ=0.0841 which can be seen as baseline for the following tests. C. Results In an evaluation which is run on each decision distribution in the dataset, the best parameter value for the reconstruction of a model is estimated using Equation 5 with PSNR as similarity measure. After that the parameter value is checked against the decision distribution with Definition 1. Therefore if theparametervalue is insideµ±σ of thehumandecisions, the parameter value estimation is counted as correct and false otherwise. This procedure lead to an estimation accuracy of 1 on the dataset of 31 judged reconstructions. The evaluation has also been run with two other (dis-) similarity measures: Root-Mean-Squared Error (RMSE) and Structural Similarity index (SSIM), see Table I. The first is a standard measure for deviations. Applying it the accuracy drops to 0.9032. This is further interesting since the RMSE is also the denominator in Equation 2. One 1The full dataset can be requested by emailing the main author. 0 1 2 3 4 5 6 7 ColorCorrection Parameter Value 0 5 10 15 20 (a) ParametertypeColourCorrection on Model 0001. 0 1 2 3 4 SmoothnessTerm Parameter Value 0 5 10 15 20 (b) Parametertype SmoothnessTerm on Model 0093. Fig. 2. Examples of human decision distributions for parameter-types ColourCorrection on Model 0001 in Subfigure (a) and SmoothnessTerm on Model 0093 in Subfigure (b). One can see the variance in the data. (Dis-)similarity Accuracy PSNR 1 RMSE 0.9032 SSIM 0.9032 TABLE I ACCURACY ON THE DATASET EVALUATED WITH DIFFERENT IMAGE (DIS-)SIMILARITIES FOR FORMULA 4. EVALUATED MEASURES ARE PEAK SIGNAL TO NOISE RATIO (PSNR), STRUCTURAL SIMILARITY INDEX (SSIM) AND ROOT-MEAN-SQUARED ERROR (RMSE). PSNR PERFORMS BEST. can see that the logarithm in the equation is important in this context. When applying SSIM, which should reflect human per- ception, the accuracy drops to 0.9032. A detailed look on the results reveals that RMSE as well as SSIM fail on the models assigned toColourCorrection. A similar comparison has been performed with different colour spaces, see Table II. Beside the HSV colour space Red, Green, Blue (RGB), YCbCr, Grayscale and CIE-Lab colour spaces have been evaluated. RGB is a standard in image representation. When using it the accuracy drops to 0.7742. Recent publications indicate that YCbCr colour space shows advantages in skin detection, see Shaik et 117