Page - 69 - in Short-Term Load Forecasting by Artificial Intelligent Technologies

Energies2018,11, 3433 2.5. Three-StepRegularizationProcess The samplingofAMIused in this study is three-times larger than that of conventionalAMI, whichcollectsdataat15-min intervals. Inaddition,as the loadprofile isdecomposed intosub-profiles, the sub-profiles that havedetailed frequency characteristics canbe learnedas the input variables, but thenumberof inputvariables increases. As thenumberof inputvariables increases, the curse ofdimensionalitydegrades the learningabilitybecause thenumberofhiddennodes increases.Asa result, thenumberofhiddenlayers is increasedtosolve thecurseofdimensionality,but thiscauses thevanishinggradientproblem.Moreover,without featureselection,overfittingoccurs. The learned hypothesismayfit the trainingsetverywell,but it cannotbeextendedtonewsamples. Inaddition, without thenormalizationprocess, thecovariateshiftproblemdegradesperformance. Thecovariate shift,whichrefers to thechange in thedistributionof the inputvariablespresent in the trainingand testdata, shouldbeprevented. Therefore, theproposedmethodincludesa three-stepregularization tosolveeachof theabove-mentionedproblems. First, thedelay factorofweeklydata isestimated. Althougha largeamountofdatacanbebeneficial fordeep learning, thedistantpastdatacanresult in overfittingproblemsandincrease thecomputationtime.Asimilarproblemwasaddressed in[36] to solve thedependenceondistanthistoricaldata. In [36], adecayfactorwasusedtosolve the long-term dependencyproblemofNARX-RNN. In thispaper, theweeklydecay’sexponentbyafactor isproposedasEquation(3): Dp=2−(p−1) (3) where p is thenumberofweeks,whichgiveshighweights tonearbyweeklydata in timeandlower weights todistantweeklypastdata. Secondly, theseparatedIMFsignals (xpk,t)arenormalizedagainst theoriginal signal size (x p t ). This isbecause thesesignalscorrespondtoresidualnoisesuchas frequencies thatare toohighor too low tobe identifiedinacertainpattern. TheIMFnormalizationprocess isperformedto identify features thatdegrade learning. TheIMFnormalizationfactorgivenbyEquation(4)andT′ is thenumberof samplesof theweeklydata. Nk= ∑T ′ t=1x p k,t/T ′ ∑T ′ t=1x p t/T′ (4) Finally, as the number of hidden layers increases, the internal covariance can be shifted. The internalcovariateshift causes thedistributionof the trainingsetandtest set todiffer,whichcan leadtolocalpoints. Batchnormalization(BN)isusedtoaddress internalcovariateshift. BNnormalizes theoutputof apreviousactivation layerbysubtracting thebatchmeananddividingby thebatch standarddeviation. TheadvantagesofBNare (1) fast learning, (2) lesscareful initialization,and(3)a regularizationeffect. BNisoneof theregularization techniquesused in thedeep learningfield[41,42]. Theregularizationprocesscontributes to theaccuracyof the loadforecastingandtheoptimization of themodelbyapplyingahighweight to the inputdatahavingthemostdefiniteperiod, reducing dependencyonthepastdistantdataandavoidingthecovariateshiftof thedatagroup. The three-step regularizationprocess increases theaccuracyof the loadforecastingbyminimizingproblemsthatcan occurwhenseveral inputsare learned. 3.DeepLearning Deep learning is one of themachine learning techniques that proposes tomodel high-level abstractions indatabyusingANNarchitecturescomposedofmultiplenon-linear transformations. Deep learning refers to stacking multiple layers of neural networks and relying on stochastic optimizationtoperformefficientmachine learningtasks. Totakeadvantageofdeeplearning, three technical constraintsmustbesolved. Thethree technical constraintsare (1) the lackofsufficientdata, (2) the lackofcomputingresources fora largenetworksize, and(3) the lackofanefficient training algorithm.Recently, theseconstraintsweresolvedbythedevelopmentofbigdataapplications, the 69