Page - 183 - in Short-Term Load Forecasting by Artificial Intelligent Technologies

Energies2018,11, 2008 While theVitullomodelandother linearmodelsperformwellon linearstationarytime-series, theyassume that loadhas roughlya linearly relationshipwith temperature [7]. However, natural gas demand time series is not purely linearwith temperature. Someof the nonlinearities can be modeledusingheating and coolingdegrees, but natural gas demandalso containsmany smaller nonlinearities thatcannotbecapturedeasilywith linearorautoregressivemodelsevenwithnonlinear transformationsof thedata. Toaddress thesenonlinearities, forecastershaveusedartificialneuralnetworks (ANNs) inplace of, or in conjunctionwith, linearmodels [5,8,9]. ANNsareuniversal approximators,meaning that with therightarchitecture, theycanbeusedtomodelalmostanyregressionproblem[8]. Artificial neuralnetworksarecomposedofprocessingnodes that takeaweightedsumof their inputsandthen outputanonlinear transformof thatsum. Recently, new techniques for increasing thedepth (number of layers) ofANNshaveyielded deep neural networks (DNN) [10]. DNNshave been applied successfully to a range ofmachine learningproblems, includingvideoanalysis,motioncapture, speechrecognition,andimagepattern detection[10,11]. As will be described in depth in the next section, DNNs are just large ANNs with the main difference being the training algorithms. ANNs are typically trained using gradient descent. Large neural networks trained using gradient descent suffer from diminishing error gradients.DNNsare trainedusingthecontrastivedivergencealgorithm,whichpre-trains themodel. Thepre-trainedmodel isfine-tunedusinggradientdescent [12]. Thismanuscriptadapts theDNNstoshort-termnaturalgasdemandforecastingandevaluates DNNs’performanceasa forecaster. Littleworkhasbeendone in thefieldof timeseries regression usingDNNs, and almost nowork has been done in the field of energy forecastingwithDNNs. Onenotableexampleof literatureonthesesubjects isQuietal.,whoclaimtobe thefirst touseDNNs for regressionand timeseries forecasting [13]. They showpromising results on three electric load demand time series and several other time series using 20DNNsensembledwith support vector regression.However, theDNNstheyusedwerequitesmall; the largestarchitectureconsistsof two hiddenlayersof20neuronseach. Becauseoftheirsmallnetworks,Quietal. didnottakefulladvantage of theDNNtechnology. Another example of work in this field is Busseti et al. [14], who found that deep recurrent neuralnetworks significantlyoutperformed theotherdeeparchitectures theyused for forecasting energydemand.Theseresultsare interestingbutdemonstratedpoorperformancewhencompared to the industrystandardinenergyforecasting,andtheyarenearly impossible toreplicategiventhe information in thepaper. Somegoodexamplesof timeseries forecastingusingDNNsincludeDalto,whousedthemfor ultra-short-termwindforecasting[15], andKuremotoetal. [16],whousedDNNsontheCompetition onArtificial Time Series benchmark. In both applications, DNNsoutperformedneural networks trainedbybackpropagation. Dalto capitalizedon theworkofGlorot andBengiowhendesigning hisnetworkandshowedpromisingresults [17].Meanwhile,KuremotosuccessfullyusedKennedy’s particle swarmoptimization inselecting theirmodelparameters [18]. Theworkmostsimilar toours is Ryuetal.,whofoundthat twodifferent typesofexaminedDNNsperformedbetteronshort-termload forecastingofelectricity thanshallowneuralnetworksandadoubleseasonalHolt-Wintersmodel [19]. Other, more recent examples of work in this field include Kuo and Huang [20], who use a seven-layer convolutional neural network for forecasting energy demand with some success. Unfortunately, theydonotuseanyweatherinformationintheirmodelwhichresultsinpoorforecasting accuracycomparedto thosewhodoaccount forweather. Lietal. usedaDNNcombinedwithhourly consumptionprofile informationtodohourlyelectricitydemandforecasting[21].Chenetal. useda deepresidualnetworktodobothpointandprobabilisticshort-termloadforecastingofnaturalgas[22]. Perhaps themostsimilar recentworkto thatwhich ispresentedinthispaper isHoseinandHosein, whocomparedaDNNwithoutRBMpretraining toonewithRBMpretrainingonshort-term load 183