Page - 136 - in Document Image Processing

J. Imaging 2018,4, 15 Figure6.CRNNsystemarchitecture. TheAdamoptimizer [38]wasused to train thenetworkwith the initial learning rateof 0.001. This algorithmcouldbe thought of as anupgrade forRMSProp [39], offeringbias correction and momentum [40]. It provides adaptive learning rates for the stochastic gradient descent update computedfromthefirstandsecondmomentsof thegradients. It alsostoresanexponentiallydecaying averageof thepast squaredgradients (similar toAdadelta [41]andRMSprop)andthepastgradients (similar tomomentum). Batchnormalization,asdescribed in [42],wasaddedaftereachconvolutional layer in order to accelerate the trainingprocess. It basicallyworks bynormalizing each batch by both themean and variance. The networkwas trained in an end-to-end fashionwith the CTC loss function[35]. 136