Page - 95 - in Proceedings of the OAGM&ARW Joint Workshop - Vision, Automation and Robotics

Feature Rank R˜ank Recall@100 Recall@500 MAP LBP 230,784.8 0.124 0.267 0.323 0.178 LBP (RotInv) 250,123.1 0.135 0.305 0.389 0.164 Shapeme 201,853.2 0.10856828071845802 0.488 0.513 0.378 Centrist 307,558.3 0.165 0.500 0.502 0.496 BPP 327,727.0 0.176 0.500 0.503 0.496 JCD 267,515.1 0.144 0.503 0.512 0.492 ACCID 227,305.3 0.122 0.505 0.510 0.499 PHOG15 220,036.4 0.122 0.5248344370860927 0.5364238410596026 0.5157031013278772 TABLE I THE RESULTS OF THE STRICT GROUND TRUTH (302 QUERIES) EVALUATED ON THE FULL USTTAB COLLECTION IN TERMS OF AVERAGE RANK, NORMALIZED RANK, RECALL AT 100, RECALL AT 500, AND MEAN AVERAGE PRECISION. str ict ma jor ity mi no rit y0.49 0.5 0.51 0.52 0.53 PHOG ACCID JCD BPP Centrist str ict ma jor ity mi no rit y0.35 0.36 0.37 0.38 0.39 Shapeme Fig. 6. Comparison of MAP results of different algorithms on the three USTTAB ground truths strict, majority and minority for the full collection. The x axis is scaled to represent the number of queries in each ground truth (302 for strict, 750 for majority and 882 for minority). While BPP, ACCID, JCD and CENTRIST hardly show any change in value, PHOG and Shapeme seem to mirror the human perception. 0 1 2 3 4 5 6 7 8 49 Fig. 7. Examples of retrieval results for a logo pair from the ground truth. At rank 0 the image shows the query, then the first eight results and only at rank 49 the logo from the corresponding USTAB trial. aim to take a close look at the evaluation procedure, ie. by investigating the possibility of taking into account similar images that have not been in trials, as has been done for the pooling method in text information retrieval [18]. The data set has already been employed for testing dif- ferent parameters of the PHOG and Shapeme features as well as extensive evaluations using other local and global features alike. The findings have already been integrated in the trademark search engine of the World Intellectual Property Organization (WIPO) of the United Nations6. However, there is a long way to go and there are several tasks, for which we propose crowd workers to be employed: Identification of multiple instances. As noted before 6http://www.wipo.int/branddb, last visited 2016-08-30 logos are submitted and re-submitted by the same company all around the world. These duplicate entries, which are often near duplicates in the visual domain, are visually similar, but should be considered separately. Crowd workers could identify and label the (near) duplicate entries. Offending logos not investigated by the appeal board. As it is a lengthy and complicated process to file an appeal, there are a lot of visually confusing similarities that have no been investigated by the appeal board. In the current version of the data set these offending logos might show up as false positives in benchmarking. Crowd workers could label the offending logos to be treated separately. Judging visually confusing similarity. While we had experts judge the offending logos upon visual vs. conceptual confusion, we think that the intuitive concept of visually confusing logos in the head of actual consumers is different to the concept adopted by legal experts. With the help of crowd workers we could paint a picture of how consumers see visual trademarks as well as the relevance and impact of offending logos and provide feedback to the legal experts. 95