Time-varying reinforcement learning for robust adaptive estimator design with application to HVAC flow control

1. A computer-implemented method using a reinforcement learning trained reduced order estimator (RL-trained ROE) and a closure model for controlling a heating, ventilation, and air conditioning (HVAC) system including actuators, wherein the method uses a processor coupled with a memory storing instructions implementing the method, wherein the instructions, when executed by the processor, carry out at steps of the method, comprising:

acquiring, via an input interface, setpoints of the HVAC system from a user input and measurement data from sensors arranged in the HVAC system;

computing a high-dimensional state estimate using the measurement data and an estimate of reduced-order state from the RL-trained ROE;

determining a controller with respect to the setpoints by using the RL-trained ROE;

generating control commands corresponding to the computed high-dimensional state estimate, based on the controller; and

transmitting, via an output interface, the control commands including instructions that control operations of the actuators of HVAC system.

2. The method of claim 1 , wherein the controller is designed using a model predictive control.

3. The method of claim 1 , wherein the controller is designed using Lyapunov design.

4. The method of claim 1 , wherein the controller is designed using robust control that takes into account model uncertainties in the RL-trained ROE.

5. The method of claim 1 , wherein the RL-trained ROE controller is trained using Proximal Policy Optimization (PPO) algorithm.

6. The method of claim 1 , wherein the RL-trained ROE is trained using Trust Region Policy Optimization (TRPO) algorithm.

7. The method of claim 1 , wherein the RL-trained ROE is trained using Robust Constrained Markov Decision Process (RCMDP) algorithm.

8. The method of claim 1 , wherein the RL-trained ROE is trained using time-varying non-stationary MDP.

9. An apparatus for controlling a heating, ventilation, and air conditioning (HVAC) system including actuators, comprising:

an input interface configured to acquire setpoints of the HVAC system from a user input and measurement data from sensors arranged in the HVAC system;

at least one memory configured to store instructions implementing a computer-implemented method; and

at least one processor coupled with the at least one memory, wherein the instructions, when executed by the at least one processor, carry out at steps of the computer-implemented method, comprising:

computing a high-dimensional state estimate using the measurement data and an estimate of reduced-order state from the RL-trained ROE;

determining a controller with respect to the setpoints by using the RL-trained ROE; and

generating control commands corresponding to the computed high-dimensional state estimate, based on the controller; and

an output interrace configured to transmit the control commands including control instructions that control the actuators operating the HVAC system.

10. The apparatus of claim 9 , wherein the controller is designed using optimal control.

11. The apparatus of claim 9 , wherein the controller is designed using Lyapunov design.

12. The apparatus of claim 9 , wherein the controller is designed using robust control.

13. The apparatus of claim 9 , wherein the RL-trained ROE is trained using Proximal Policy Optimization (PPO) algorithm.

14. The apparatus of claim 9 , wherein the RL-trained ROE is trained using Trust Region Policy Optimization (TRPO) algorithm.

15. The apparatus of claim 9 , wherein the RL-trained ROE is trained using Robust Constrained Markov Decision Process (RCMDP) algorithm.

16. The apparatus of claim 9 , wherein the RL-trained ROE is trained using time-varying non-stationary MDP.