Seite - 198 - in Document Image Processing

J. Imaging 2018,4, 32 therecall constraint. Figure10depicts theuser interfaceofourevaluation toolaswellas theprecision andrecallcurves,wherex-axisdenotes trvaluesandy-axisdenotes tpones. Theproposedperformance metrics and theirunderlyingconstraints are similar to thoseused in ICDAR2013 [24] and ICDAR 2015[25]RRCs. It isworthnotingthatourannotationandevaluationtoolsare fully implementedin Javaandaremadeopen-source forstandardizationandvalidationpurposes. Frame list Frame preview Evaluation buttons a b Figure10.AcTiV-Devaluation tool. Theusercanapply theevaluationprocedure to thecurrent frame “EvaluateCF”buttonor toallvideoframes“EvaluateAll”button(a). The“PerformanceValue”button displaysprecision, recallandF-scorevalues (b). 4.2. RecognitionProtocols andMetrics Table5depicts therecognitionprotocols. • Protocol3aimstoevaluate theperformanceofOCRsystemstorecognize texts inHDframes. • Protocol6 is similar toProtocol3,differingonlyby thechannel resolution. AllSD(720×576) channels inourdataset canbe targetedbythisprotocolwhich is split in foursub-protocols: three channel-dependent (Protocols6.1,6.2and6.3)andonechannel-free (Protocol6.4). • Protocol 6bis is dedicated to the newadded resolution (480× 360) for the TunisiaNat1 TV channel. Themain ideaof thisprotocol is to trainagivensystemwithSD(720×576)data i.e., Protocol6.3andtest itwithdifferentdataresolutionandquality. • Protocol9 is thegenericversionofProtocols3and6where text recognition isassessedwithout consideringdataquality. Table5.RecognitionEvaluationProtocols. “Lns”and“Wds”respectivelydenote“Lines”and“Words”. Protocol TVChannel Training-Set Test-Set ClosedTest-Set #Lns #Wds #Chars #Lns #Wds #Chars #Lns #Wds #Chars 3 AlJazeeraHD 1909 8110 46,563 196 766 4343 262 1082 6283 6 France24 1906 5683 32,085 179 667 3835 191 734 4600 RussiaToday 2127 13,462 78,936 250 1483 8749 256 1598 9305 TunisiaNat1 2001 9338 54,809 189 706 4087 221 954 5597 AllSD 6034 28,483 165,830 618 2856 16,671 668 3286 19,502 6bis TunisiaNat1+ - - - 320 1487 8726 311 1148 6645 9 All 7943 36,593 212,393 814 3622 21,014 930 4368 25,785 Metrics:Theperformancemeasure for therecognition task isbasedontheLineRecognitionRate (LRR),WordRecognitionRate(WRR)at thelineandwordslevels, respectively,andonthecomputation of insertion(I),deletion(D) andsubstitution(S) errorsat thecharacter level (CRR) thataredefinedas: CRR= #characters− I−S−D #characters (4) 198