Page - 70 - in Short-Term Load Forecasting by Artificial Intelligent Technologies

Energies2018,11, 3433 Internet-of-Things,andhighperformancesmartcomputing[37–39].Oneof themostefficientdeep learningprocesses isRNN. RNNsare fundamentallydifferent fromtraditional feed-forwardneuralnetworks;RNNshave a tendency to retain informationacquired through subsequent time-stamps. This characteristic of RNNsisuseful for loadforecasting. EventhoughRNNshavegoodapproximationcapabilities, they arenotfit tohandle long-termdependenciesofdata. Learning long-rangedependencieswithRNNs ischallengingbecauseof thevanishinggradientproblem.The increase in thenumberof layersand the longerpaths to thepastcause thevanishinggradientproblembecauseof theback-propagation algorithm,which has the verydesirable characteristic of being veryflexible, although causes the vanishinggradientproblem[30,32–34]. 3.1. LongShort-TermMemoryNeuralNetworks Thelong-short termmemorynetworkhasbeenemployedtoapproachthebestperformanceof state-of-the-artRNNs. Theproblemof thevanishinggradient is solvedbyreplacingnodes in theRNN withmemorycellsandagatingmechanism. Figure2showstheLSTMblockstructure. Theoverall support inacell isprovidedbythreegates. Thememorycell state st−1 interactswith the intermediate output ht−1. The sub-sequent input xt determineswhether to remember or forget the cell state. The forgetgate ftdetermines the input for thecell state st−1 using thesigmoidfunction. The input gate it, inputnodegt, andoutputnodeotdetermine thevalues tobeupdatedbyeachweightmatrix, whereσ represents thesigmoidactivationfunction,whileφ represents the tanhfunction. Theweight matrices in theLSTMnetworkmodelaredeterminedbytheback-propagationalgorithm[37–42]. TheLSTMhasbecomethestate-of-the-artRNNmodel foravarietyofdeep learningtechniques. Severalvariantsof theLSTMmodel for recurrentneuralnetworkshavebeenproposed.VariantLSTM models havebeenproposed to improveperformanceby solving issues suchas computation time andthemodelcomplexityof thestandardLSTMstructure.Amongthevariants, theGRUmaintains performancebysimplifyingthestructurewithanupdategate that iscoupledwithaninputgateand forgetgate. Thestructureof theGRUisadvantageousfor forecasting ina large-scalegrid toreduce calculation time[42]. In [45],variantsof theLSTMarchitectureweredesignedandtheirperformances werecomparedthroughimplementation. Theresults revealedthatnoneof thevariantsofLSTMcould improveuponthestandardLSTM.Inotherwords,aclearwinnercouldnotbedeclared. Therefore, thepopularLSTMnetworksareusedin this study[45,46]. Figure2.Thestructureof theLSTM. 3.2.NonlinearAutoregressiveNetworkwithExogenous Inputs NARXRNNsandLSTMsolve thevanishinggradientproblemwithdifferentmechanisms.NARX RNNsallowdelays fromthedistantpast layer,but this structure increasescomputationtimeandhasa smalleffectonlong-termdependencies. TheLSTMsolves thevanishinggradientproblembyreplacing nodes in theRNNwithmemorycellsandagatingmechanism[36]. 70