Page - 93 - in Proceedings of the OAGM&ARW Joint Workshop - Vision, Automation and Robotics

Fig. 2. The view on a pair of logos in the visual similarity evaluation application. II. DATASET AsalreadymentionedGoogleoffers several trademarkcol- lections as free download4 in cooperation with the USPTO. Note at this point that these downloads offer the actual USPTO data, ie. the actual image files filed for registration as well as the resulting metadata. On the Google site, daily trademark applications, images and the USTTAB trials data from 1955 until today are available. All of these can be downloaded inchronologicallyorderedZIP-archivescontain- ing an XML file with describing all trials in the specific period of time. A. Selection Criteria For the creation of our new ground truth, the trials from 1955 until end of August 2015 were chosen, being all trials available at the time of extraction. Each trial entry in the retrieved data contains the party-information, a section that includes information about all parties involved in the trial. Each party has zero or more properties, which correspond to the trademarks associated with it. The properties are identified by a unique identification and a serial number. A first filtering step was taken by selecting only those trials that do regard an opposition. For the dataset only op- positions are interesting, as those contain cases of confusing similarities, in contrast to obvious ones. In the next step, all entries with exactly two parties and exactly one associated property per party were selected. In all other cases it is not possible to distinguish the trademarks relevant for this claim. By joining this data with all available US trademark images, trials regarding non-visual trademarks could be removed. As the presence of trademark images does not guarantee that the trial was filed because of visual similarity, the next 4https://www.google.com/googlebooks/ uspto-trademarks.html, last visited 2016-01-19 0 1 Fig. 3. One of the logo pairs in the USTTAB strict ground truth step was to detect the type of similarity. Unfortunately, there is no formal classification contained in the data. To overcome this problem, a web-based application was developed, which allows experts to decide whether the trial was based on visual similarity or not. The experts were chosen from three different areas of expertise: One from the field of visual information retrieval at the University of Klagenfurt, one from the field of trademark retrieval at the World Intellectual Property Organisation and one with appropriate knowledge in both fields. To be able to create a sufficiently big ground truth in reasonable time, 1000 trials were randomly chosen from the previously selected. The application showed two trademark images next to each other and asked the expert to decide whether the claim was due to visual similarity or not. To assist the experts in their decision, the trademark name was presented beyond the image if one was present (compare Fig. I). For the 1000 logo pairs, all experts agreed on visual similarity in 160 cases. At least two of the three experts agreed on visual similarity in 384 cases while there are 451 trials inwhichonlyoneexpert judged that the trialwasdue to visual similarity. The 1000 pairs included nine control pairs of obvious visual similarity, which were correctly answered by all experts. B. Properties The resulting dataset consists of 1.8 million visual trade- marks. Those trademarks are either registered, pending or canceled in the USPTO registration data base. The set is composed by 1,587,248 verbal signs, 533,910 non-verbal signs and 4,867,626 combined trademarks. The signs are of varying image quality with different resolution, in color, gray scale or binary black & white format. As this data is directly form the USPTO’s registration data base, its composition is realistic and, therefore, well suited for objective evaluations. From the USTTAB trials and the expert’s decisions, three blends of the data set were created. The first blend includes only logosonwhichall expertsagreed. It is therefore referred to as strict ground truth. An example for this set can be seen in Fig. 3. The second blend consists of the logo pairs a majority of experts agreed on, the majority ground truth (cp. Fig. 4). Finally, the minority ground truth consists of all pairs with at least one expert voting for visual similarity (cp. Fig. 5). 93