Page - 92 - in Document Image Processing

J. Imaging 2018,4, 41 itdividesthegradientbysquarerootofthemeansquarevalue. Thefirstmovingaverageofthesquared gradient isgivenby, AvtγAvt−1+(1−γ)(∇Qw)2 (6) whereγ is theforgettingfactor,∇QwisthederivativeoftheerrorandAvt−1 isthepreviousadjustment value. Theweightsareupdatedasper followingequation, wt+1wt− α√Avt ∇Qw (7) wherew is thepreviousweightandwt+1 is theupdatedweightwhereasα is theglobal learningrate. Adam(adaptivemomentestimation)[30] isanotheroptimizerforDCNNthatneedsthefirst-order gradient with small memory and computes adaptive learning rate for different parameters. Thismethodhasprovenbetter thantheRMSpropandrpropoptimizers. Therescalingof thegradient isdependentonthemagnitudesofparameterupdates. TheAdamdoesnotneedastationaryobject andworkswithsparsegradients. It alsocontainsadecayingaverageofpastgradientsMt. Mt=B1Mt−1+(1−B1)Gt (8) Vt=B2Vt−1+(1−B2)G2t (9) whereMt andVt are calculatedfirst and the secondmomentof thegradients and thesevaluesare biasedtowardszerowhenthedecayratesaresmall, andtherebybias-correctionhasdonefirstand secondmomentsestimates: Mˇt= Mt 1−Bt1 (10) Vˇt= Vt 1−Bt2 (11) AspertheauthorsofAdam,thedefaultvaluesofB1 andB2werefixedat0.9and0.999empirically. Theyhaveshownitswork inpracticeasabest choiceasanadaptive learningmethod. Adamax is anextensionofAdam,where inplaceofL2 norm,anLPnorm-basedupdaterulehasbeenfollowed. 3.3. LayerwiseTrainingDCNNModel Theworkof training is tofindthebestweight for thedeepneuralnetworkatwhich thenetwork produceshighaccuracyoraverysmallerror rate. Theoutcomeofanydeepmodelneuralnetwork somehowdepends onhow themodelwas trained and thenumber of layers. Usually, themodel is createdwith the certain number of layers, and entire layers are being involved in the training phase. In thiswork,weproposeda layer-wise trainingmodelofDCNNinspiteof involvingentire layersduringthe trainingphase torecognize thehandwrittenDevanagari characters. The layer-wise trainingmodel startswith addingone layer of convolutional andpooling layer, followedby fully connected layer and applies the back-propagation algorithm to find the weights. In the next phaseof the layer-wise trainingmodel, thenext layerof convolutional,pooling layer isaddedand thebackpropagationalgorithm is appliedwithpreviously foundweights to calculateweights for theaddedlayer. After addingentire layers, afine tuningwasperformedwith the completenetwork to adjust theentireweightsof thenetworkonavery lowlearningrate. Theback-propagationalgorithmstarts withsomerandomweights,andduringtrainingitsharpenstheweighsbyupdatingthemineachepoch. The layer-wise trainingmodelprovidesniceroughweights initiallyas thenetworkstartswithfirst layersand, further, it addsremaining layers tofindtheweights for remaining layers. The layer-wise trainingmodel is clearlyshowninFigure2. The trainingstartswithonlyonepairofconvolutional andpooling layerandfurtheranotherpair isbeingadded.Algorithm1showsthestepwiseprocedure tocreate the layer-wiseDCNNmodel. 92