Page - 121 - in Adaptive and Intelligent Temperature Control of Microwave Heating Systems with Multiple Sources

4.2. IntelligentControl control policy used in AC methods is a parameterized family. The parameters of the control policy are updated either directly using the TD error, such as the Gibbs softmax algorithm [BV03] [PL05] and the adaptive heuristic critic learning architecture (AHCON) [Lin93], or using the gradient term that is influenced by the critic, like the policy gradient method [SMSM99] and the natural policy gradientmethod[PVS05] [PS06] [PS08a]. PlantController (Actor) Critic Current State S New State S Control Action U Reward Reward R Figure4.9. Actor-critic control system[GBLB12]. Thestructureof theACcontrolsystemseemssimilarwiththecon- ventional feedback controller (such as figure 4.2). To be more spe- cific, the principle of AC methods is very close to the semi-direct SPSA based NN controller proposed previously (see figure 4.6). Despiteminordifferenceswithrespecttodifferentformsofrealiza- tions, both controllers are updated using terms either from direct interactions with the plant (TD error or direct SPSA) or gradients approximated from direct interactions (gradient methods or indi- rect SPSA). The critic use in AC methods can be represented by a lookup table such as inQ-learning or sarsa, but it can also be ap- proximatedusingeitherlinearornonlinear(suchasNN)functions [BM95] [Sut96] [SK00], which is similar with the system estimator in semi-direct SPSA NN control system. From another aspect, AC methodscanalsobeconsideredas thecombinationofacritic-only algorithmandamodel-freecontroller [LV09]. OneadvantageofACmethodsoverclassicalQ-learningandsarsa is that it is able to deal with continuous states and actions. Both the control policy and the value function in AC methods can be 121