Deep reinforcement learning-based nonlinear control for magnetic levitation systems of maglev trains

Abstract

Magnetic levitation (maglev) train as a novel transportation can offer various advantages including non-contact operation, minimal vibration, low noise, no risk of derailment, flexible route selection, and cost-effective construction. However, the levitation for the electromagnetic suspension system (EMS) of the maglev train is accomplished based on the magnetic attraction force between electromagnets and guideway. When considered without feedback control, the maglev train system exhibits inherently unstable open-loop dynamics. Furthermore, the maglev train system possesses complex, nonlinear dynamic characteristics. External disturbances, such as wind and variations in passenger load, can also impact the system during operation. These problems can seriously affect the stability and reliability of the maglev train and may lead to partial levitation-point failure. Designing appropriate controllers for the maglev train system is a pivotal problem for maglev train. In recent years, although the control of maglev system has obtained great achievement, there still exists some drawbacks: (1) The maglev system for controller design is commonly linearized, and lack in automatic adjustment of control strategies. (2) Uncertainty in maglev train system modelling. (3) Most existing control methods can not guarantee the safe boundary for the controller. (4) Existing researches neglected the effect of crosswinds when designing controllers. (5) Lack of methods for considering coupling effect between two levitation points at one side of the bogie. In this dissertation, the EMS-type maglev control system of the maglev train is chosen as the research object. Based on the problems and drawbacks mentioned, further researches have been carried out to solve the levitation problems with a novel method named deep reinforcement learning (DRL) in this chapter. Main work for this chapter have been listed as follows: 1. Transfer learning-based DRL (TL–DRL) is proposed to develop an adaptive nonlinear levitation system controller that enables automatic adjustment of control strategies. First, levitation control based on DRL is mathematically modeled using Markov decision processes, and the nonlinear state space of a single electromagnet levitation control system is established as an agent–environment interaction with the developed deep reinforcement learning controller. Then a twin delayed deep deterministic policy gradient algorithm in an actor–critic framework is adopted to solve the Markov decision processes. To address the dispersion caused by nonlinear suspension control, a transfer learning-based two-stage training process is devised that first trains the twin delayed deep deterministic policy gradient networks on a linearized model and then transfers the networks to a nonlinear model. The effectiveness of the new controller is verified by comparing it with a conventional proportional–integral–derivative (PID) controller and an adaptive sliding mode controller. The robustness of the TL–DRL controller is examined in the presence of uncertainty, such as train load changes and disturbance forces in the suspension system. 2. Considering the coupling effect between two levitation units of the levitation bogie, a cooperative levitation controller based on the Hamilton-Jacobi-Bellman incorporated multi-agent reinforcement learning (HJB–MADRL) is proposed. The MADRL is adopted for the two-point levitation control considering the coupling effect between the two levitation points. To improve the training of the value network in the MADRL, the HJB function is used in control theory to evaluate the optimality of the value function. The proposed algorithm shows an improved performance compared to the original MADRL algorithm. The effectiveness of the proposed cooperative controller using the proposed algorithm is verified by comparing with a conventional PID controller and a model-guided controller. The robustness of the HJB–MADRL controller is examined in the presence of pitch motion, change in train load, disturbance force, and track irregularity. 3. To ensure the stability and safety of the air gap between the train and its guideway, a safe deep reinforcement learning (SDRL) controller for the maglev system considering the deformation of the flexible guideway is proposed. Notably, a reciprocal control barrier function (RCBF) is augmented in the reward function of the DRL to ensure safety and optimality of the controller. Additionally, a damping coefficient is incorporated into the designed RCBF to specify the trade-off between safety and optimality. The improved performance of the proposed SDRL is verified by comparing to original DRL algorithm. The superiority of the proposed controller is validated through a comparative analysis with a traditional PID controller and a genetic algorithm tuned super twisting sliding mode controller (GA–ST–SMC) via simulations. Additionally, the robustness of the proposed controller is assessed under conditions of changing train loads, load fluctuations, external disturbances, and track irregularities. Furthermore, experiments have also been conducted to validate the control performance of the proposed RCBF–SDRL controller in comparison to the PID controller on a magnetic levitation system. 4. To investigate the impact of crosswinds on maglev trains, a numerical model is constructed in ANSYS Fluent Meshing, considering the complexities of the environment. Validation of this numerical model is conducted through a wind tunnel test. Subsequently, the principles of fluid mechanics similarity are employed to scale wind forces to a real-world maglev train scenario. To reduce the wind effect on the maglev train, a safe deep reinforcement learning (SDRL) controller is adopted to adjust the control signal for the maglev train–guideway coupling system. Notably, a reciprocal control barrier function (RCBF) is augmented in the reward function of the DRL to ensure safety and optimality of the controller. The superiority of the proposed controller in terms of efficiency and accuracy is validated through a comparative analysis with a traditional PID controller under varying crosswind speeds and train speeds. Finally, the future work plans are presented.