Page - 122 - in Adaptive and Intelligent Temperature Control of Microwave Heating Systems with Multiple Sources

4. ControlSystemDesign generalized in the continuous states and actions spaces by using various functionapproximationmethods [SK00], respectively. After its release in the 1980s, TD immediately became one of the most popular RLC methods. Until now many improvements have been done based on conventional TD methods (as introduced above). For instance, that function approximation based approaches have been proposed to extend the conventional critic-only methods into more general methods which can deal with continuous states and actions, such as the CMAC basedQ-learning [SSR97] or the wire-fiited NN basedQ-learning [GWZ99]. More detailed introductions about TD learningmethodsrefer to [Boy02]and[Si04]. 4.2.2. DesignofReinforcementLearningController Although TD methods have been widely implemented in various ap- plications,mostofsystemsbeingcontrolledhavesimplearchitectures, which means they have either discrete states and actions, or low in- put and output space dimensions. When a TD learning controller is used to deal with complex systems with both continuous variables and high dimensions (such as HEPHAISTOS), the control system be- comes much more complicated and in most cases the corresponding learning process will become extremely slow. It is acceptable if the controllercanbetrainedwithexperimentaldataepisodesintheoffline form. However, in HEPHAISTOS it is impossible to generate training data which can comprehensively cover all dynamics of the plant. In this case, the TD learning control system has to be specially designed andoptimized. Generally it isdevelopedandimplementedaccording to the followingprinciples. Hybridmulti-agentcontrolstructure The hybrid multi-agent control structure is shown in figure 4.10. The entire control system consists of two independent controllers that are designed for different control objectives. One is a conventional adap- tivecontroller thatcoulduseeitherMPCorNNC,andtheotheroneis a lookup table based TD learning controller which will be introduced 122