Page - 375 - in Short-Term Load Forecasting by Artificial Intelligent Technologies

Energies2018,11, 1138 wherew is theweight; a is thesumcalculated throughweights; f is theactivation function; s is the valueaftercalculationbytheactivationfunction; t represents thecurrent timeof thenetwork; i is the numberof inputvectors; h is thenumberofhiddenvectors in t is time; h′ is thenumberofhidden vectors in t−1 time;ando is thenumberofoutputvectors. Figure1.AsimpleRNNstructure,whereX is the inputunit,H is thehiddenunit,Y is theoutputunit, andW is theweightmatrix. Similar to conventional neural networks, RNNscanbe trainedbyback-propagation through time[35]with thegradientdescentmethod. AsshowninFigure1, eachhidden layerunit receives notonly thedata inputbut also theoutputof thehidden layer in the last timestep. The temporal informationcanberecordedandput into thecalculationof thecurrentoutputso that thedynamic changingprocesscanbe learnedwith thisarchitecture. Therefore,RNNsarereasonable topredict the customer loadcurves inpowergrids.However,whenthe timesequence is longer, the informationwill reduceanddisappeargradually throughtransferring inhiddenunits. TheoriginalRNNshave the vanishinggradientproblemandtheperformancedeclineswhendealingwith longtimesequences. The vanishing gradient problem can be solved by adding control gates for remembering information in theprocessofdata transfer. InLSTMnetworks, thehiddenunitsofRNNsarereplaced withLSTMblocksconsistingofcell, inputgate,outputgateandforgetgate.Moreover, the forgetgate andinputgatearecombinedintoasingleupdategate inGRUneuralnetwork. ThestructureofGRU isshowninFigure2. The feedforward deduction process for GRU units is shown in Figure 2 and given by Equations (4)–(10). atu= I ∑ i=1 wiuxti+ H ∑ h=1 whust−1h (4) stu= f(a t u) (5) atr= I ∑ i=1 wirxti+ H ∑ h=1 whrst−1h (6) str= f(a t r) (7) a˜th′= s t u H ∑ h=1 whh′s t−1 h + I ∑ i=1 wih′x t i (8) s˜th′=φ(a˜ t h′) (9) sth=(1−stu)s˜th′+stust−1h (10) 375