Page - 81 - in Adaptive and Intelligent Temperature Control of Microwave Heating Systems with Multiple Sources

3.3. Black-boxModeling is the backward error propagation which computes the local gradient of each node from the output layer to the input layer. After these two roundsofcalculations,allweightscanbeupdated. Intuitively, thefor- wardpassprovidesnecessaryinformationfor thebackwardpass,and the backward pass calculates how much of the final output error be- longs to each individual weights, which will be updated accordingly. The backpropagation algorithm reduces the overall computation cost significantly and promotes the development of neural networks a big stepforward. Based on the standard online SGBP algorithm, additional strategies havetobetakentoreducetheeffectof”catastrophic interference”. As stated in [Fre99], there are several possible strategies, by either using noise-freetrainingdata,applyingmoreconstructivenetworkarchitec- tures or adding extra information storage besides pure weights of the network. Thefirst twoapproachesaredifficult to implement inrealis- ticapplications, thereforemostpracticalonlinelearningalgorithmsgo to the thirddirection, suchas the followingthreealgorithms. • BPM is also named as enhanced backpropagation [Roj96], which involves the momentum term in the weight update expression, suchas ∆w(q) =−η ∂JOL ∂w ∣∣∣∣ k +α∆w(q−1), (3.88) whereα is a tuning factor indicating the influence from the mo- mentum. The involvement of momentum brings information and influences from past training date pairs, which effectively attenu- ate weight oscillations during the iteration process and improves theestimationstability. • This idea of using EKF in neural networks learning was firstly ap- pliedin[SW89]andquicklybecameapowerfulandefficient learn- ing algorithm, especially in incremental learning mode [KV92] [RRK+92] [WVDM00]. The update equations using EKF in NN trainingissimilarwiththeproceduresdescribedinformernonlin- ear system identification (equations 3.67 and 3.68), and the corre- spondingderivationprocesscanalsobe foundin[TP08]. • Sliding window based backpropagation [CMA94] is a way to use batch learning mode in a online form. Instead of using the entire 81