Page - 398 - in Short-Term Load Forecasting by Artificial Intelligent Technologies

Energies 2018,11, 242 3.3.2. Pre-Trainingof theDBNPart Generally speaking,with thenumberof hidden layers increasing, the effectiveness of theBP algorithmforoptimizingtheparametersof thedeepneuralnetworkisgettinglowerandlowerbecause of the gradient divergence. Fortunately,Hinton et al. [11] proposed a fast learning algorithm for theDBN.Thisnovel approach realizes layer-wisepre-trainof themultipleRBMs in theDBNina bottom-upwayasdescribedbelow: Step1: Initialize thenumberofhidden layers k, thenumberof the trainingdataN andthe initial sequencenumberofhiddenlayeru=2. Step2: Assignasamplex fromthetrainingdata tobe the inputdataof theDBN. Step3: Regardthe input layerandthefirsthiddenlayerof theDBNasanRBM,andcompute the activationA1(x)byEquation(3)whenthe trainingprocessof thisRBMisfinished. Step4: Regard the uth and the (u+1)th hidden layer as an RBM with the input Au−1(x), andcompute theactivationAu(x)byEquation (3)when the trainingprocessof thisRBM iscompleted. Step5: Letu=u+1,anditerateStep4untilu> k. Step6: Use theAk(x)as the inputof theregressionpart. Step7: Assignanothersample fromthe trainingdataas the inputdataof theDBN,anditerateStep3 to7until all theN trainingdatahavebeenassigned. 3.3.3. LeastSquaresLearningof theRegressionPart Suppose that the training set is ℵ = {(x(l),y(l))|x(l) ∈ Rn,y(l) ∈ R, l = 1, · · · ,N}. As aforementioned, once the pre-training of the DBN part is completed, the activation of the final hidden layer of the MDBN with respect to the input x(l) can be obtained to be Ak(x(l)), where l = 1,2,. . . ,N. Furthermore, the activation of the final hidden layer of theMDBNwith respect toall theN trainingdatacanbewritten in thematrix formas Ak(X)= [Ak(x(1)),Ak(x(2)), · · · ,Ak(x(N))]T = ⎡⎢⎢⎢⎢⎢⎢⎣ σ ( bk+wkσ ( · · ·+w2σ ( b1+w1x(1) ))) σ ( bk+wkσ ( · · ·+w2σ ( b1+w1x(2) ))) ... σ ( bk+wkσ ( · · ·+w2σ ( b1+w1x(N) ))) ⎤⎥⎥⎥⎥⎥⎥⎦ N×nk , (15) wherenk is thenumberofneuronsof thekthhiddenlayer. Wealways expect that each actual value y(l)with respect to x(l) canbe approximatedby the output yˆ(l)of thepredictorwithnoerror. Thisexpectationcanbemathematicallyexpressedas N ∑ l=1 ‖yˆ(l)−y(l)‖=0, (16) where yˆ(l) is theoutputof theMDBNandcanbecomputedas yˆ(l) =Ak(x(l))β (17) inwhichβ is theoutputweightingvectorandcanbeexpressedas β=[β1,β2, · · · ,βnk]Tnk×1. (18) Then,Equation(16)canberewritten in thematrix formas 398