Direct policy optimization for meeting room comfort control and energy management

1. A method for controlling comfort in a building zone, the method comprising:

receiving historical data, the historical data indicative of comfort and occupancy of the building zone during one or more historical states;

training a system model to represent a dynamic response of the building zone based on the historical data;

obtaining a policy function that defines a stage cost over a time horizon based on a set of predicted inputs to equipment that operate to affect the comfort in the building zone and predicted outputs indicative of the comfort in the building zone;

determining a control law by optimizing the policy function, the control law implemented as a neural network configured to process the system model, wherein determining the control law comprises determining a set of parameters that define a relationship between one or more inputs to the neural network and one or more outputs of the neural network, and wherein optimizing the policy function comprises (i) determining a value of the stage cost for a plurality of different values of the set of parameters and (ii) selecting values of the set of parameters that optimize the value of the stage cost; and

performing online control of the building zone using the control law.

2. The method of claim 1 , wherein training the system model to represent the dynamic response of the building zone comprises:

training a plurality of models, each of the plurality of models configured to represent a system response, wherein each system response uses different control schemes, wherein the plurality of models comprise system models; and

selecting the system model based on the system models responsive to determining an accuracy of the system model in representing the dynamic response of the building zone is higher than an accuracy of other models of the plurality of models.

3. The method of claim 1 , wherein determining the control law by optimizing the policy function comprises:

minimizing an average value of the policy function using a plurality of control laws each implemented as a neural network and configured to process the system model; and

selecting the control law from the plurality of control laws resulting in a lowest minimized average value associated with the policy function.

4. The method of claim 1 , wherein:

training the system model comprises training at least one of a Long Short Term Memory (LSTM) model, a Greybox model, a linear physics model, or a disturbance model, and

determining the control law by optimizing the policy function comprises optimizing at least one of a dense policy, a naïve policy, or a rule-based policy.

5. The method of claim 1 , further comprising:

performing online control of the building zone using a rule-based policy, the rule-based policy configured to receive state data of the building zone and determine comfort setpoint adjustments based on the state data; and

collecting the state data and associated comfort setpoint adjustments over a period of time to be used as the historical data;

wherein training the system model to represent the dynamic response of the building zone comprises training the system model using the collected state data and the associated comfort setpoint adjustments; and

wherein performing online control of the building zone using the control law occurs after performing online control of the building zone using the rule-based policy.

6. The method of claim 5 , further comprising:

comparing an implementation of the rule-based policy with an implementation of the control law;

determining that the implementation of the control law provides an increased level of comfort or increased level of energy savings above a predetermined threshold; and

replacing online control using the rule-based policy with online control using the control law.

7. The method of claim 1 , wherein:

training the system model comprises training the system model to determine adjustments of manipulated variables (MVs) that will approach a control variable (CV) setpoint based on the historical data, the historical data comprising sets of CVs, MVs, and disturbance variables (DVs);

the CVs comprise at least one of: the temperature of the building zone or variable air volume (VAV) airflow entering the building zone;

the MVs comprise at least one of: a heating setpoint of the building zone or a cooling setpoint of the building zone; and

the DVs comprise at least one of: ambient temperature, solar intensity, or scheduled occupancy of the building zone.

8. A controller for controlling comfort in a building zone, the controller comprising one or more processors and one or more memories storing instructions that, when executed by the one or more processors, cause the one or more processors to perform operations comprising:

receiving historical data, the historical data indicative of comfort and occupancy of the building zone during one or more historical states;

training a system model to represent a dynamic response of the building zone based on the historical data;

obtaining a policy function that defines a stage cost over a time horizon based on a set of predicted inputs to equipment that operate to affect the comfort in the building zone and a predicted outputs indicative of the comfort in the building zone;

determining a control law by optimizing the policy function, the control law implemented as a neural network configured to process the system model, wherein determining the control law comprises determining a set of parameters that define a relationship between one or more inputs to the neural network and one or more outputs of the neural network, and wherein optimizing the policy function comprises (i) determining a value of the stage cost for a plurality of different values of the set of parameters and (ii) selecting values of the set of parameters that optimize the value of the stage cost; and

performing online control of the building zone using the control law.

9. The controller of claim 8 , wherein training the system model to represent the dynamic response of the building zone comprises:

training a plurality of models, each of the plurality of models configured to represent a system response, wherein each system response uses different control schemes, wherein the plurality of models comprise system models; and

selecting the system model based on the system models responsive to determining an accuracy of the system model in representing the dynamic response of the building zone is higher than an accuracy of other models of the plurality of models.

10. The controller of claim 8 , wherein:

training the system model comprises training at least one of a Long Short Term Memory (LSTM) model, a Greybox model, a linear physics model, or a disturbance model, and

determining the control law by optimizing the policy function comprises optimizing at least one of a dense policy, a naïve policy, or a rule-based policy.

11. The controller of claim 8 , wherein determining the control law by optimizing the policy function comprises:

minimizing an average value of the policy function using a plurality of control laws each implemented as a neural network and configured to process the system model; and

selecting the control law from the plurality of control laws resulting in a lowest minimized average value associated with the policy function.

12. The controller of claim 8 , wherein the instructions further cause the one or more processors to perform operations comprising:

performing online control of the building zone using a rule-based policy, the rule-based policy configured to receive state data of the building zone and determine comfort setpoint adjustments based on the state data; and

collecting the state data and associated comfort setpoint adjustments over a period of time to be used as the historical data; and

wherein training the system model to represent the dynamic response of the building zone comprises training the system model using the collected state data and the associated comfort setpoint adjustments; and

wherein performing online control of the building zone using the control law occurs after performing online control of the building zone using the rule-based policy.

13. The controller of claim 12 , wherein the instructions further cause the one or more processors to perform operations comprising:

comparing an implementation of the rule-based policy with an implementation of the control law;

determining that the implementation of the control law provides an increased level of comfort or increased level of energy savings above a predetermined threshold; and

replacing online control using the rule-based policy with online control using the control law.

14. The controller of claim 8 , wherein:

training the system model comprises training the system model to determine adjustments of manipulated variables (MVs) that will approach a control variable (CV) setpoint based on the historical data, the historical data comprising sets of CVs, MVs, and disturbance variables (DVs);

the CVs comprise at least one of: the temperature of the building zone or variable air volume (VAV) airflow entering the building zone;

the MVs comprise at least one of: a heating setpoint of the building zone or a cooling setpoint of the building zone; and

the DVs comprise at least one of: ambient temperature, solar intensity, or scheduled occupancy of the building zone.

15. One or more non-transitory computer-readable storage media having computer-executable instructions stored thereon that, when executed by one or more processors of a control system, cause the control system to perform operations comprising:

receiving historical data, the historical data indicative of comfort and occupancy of a building zone during one or more historical states;

training a plurality of models, each of the plurality of models configured to represent a system response, wherein the plurality of models comprises a system model;

selecting the system model based on the system model responsive to determining accuracy of the system model in representing a dynamic response of the building zone is higher than an accuracy of other models of the plurality of models;

obtaining a policy function that defines a stage cost over a time horizon based on a set of predicted inputs to equipment that operate to affect the comfort in the building zone and predicted outputs indicative of the comfort in the building zone;

determining a control law by optimizing the policy function, the control law implemented as a neural network configured to process the system model, wherein determining the control law comprises determining a set of parameters that define a relationship between one or more inputs to the neural network and one or more outputs of the neural network, and wherein optimizing the policy function comprises (i) determining a value of the stage cost for a plurality of different values of the set of parameters and (ii) selecting values of the set of parameters that optimize the value of the stage cost; and

performing online control of the building zone using the control law to pre-condition the building zone.

16. The media of claim 15 , wherein using the control law to pre-condition the building zone comprises:

providing control signals to heating, ventilation, and air conditioning (HVAC) equipment to lower the temperature of the building zone prior to the building zone being occupied; or

providing control signals to the HVAC equipment to increase the temperature of the building zone prior to the building zone being occupied.

17. The media of claim 15 , wherein determining the control law by optimizing the policy function comprises:

minimizing an average value of the policy function using a plurality of control laws each implemented as a neural network and configured to process the system model; and

selecting the control law from the plurality of control laws resulting in a lowest minimized average value associated with the policy function.

18. The media of claim 15 , wherein:

training the system model comprises training at least one of a Long Short Term Memory (LSTM) model, a Greybox model, a linear physics model, or a disturbance model, and

determining the control law by optimizing the policy function comprises optimizing at least one of a dense policy, a naïve policy, or a rule-based policy.

19. The media of claim 15 , wherein the media further comprises:

performing online control of the building zone using a rule-based policy, the rule-based policy configured to receive state data of the building zone and determine comfort setpoint adjustments based on the state data; and

collecting the state data and associated comfort setpoint adjustments over a period of time to be used the historical;

wherein training the system model to represent the dynamic response of the building zone comprises training the system model using the collected state data and the associated comfort setpoint adjustments; and

wherein performing online control of the building zone using the control law occurs after performing online control of the building zone using the rule-based policy.

20. The media of claim 15 , wherein:

training the system model comprises training the system model to determine adjustments of manipulated variables (MVs) that will approach a control variable (CV) setpoint based on the historical data, the historical data comprising sets of CVs, MVs, and disturbance variables (DVs);

the CVs comprise at least one of: the temperature of the building zone or variable air volume (VAV) airflow entering the building zone;

the MVs comprise at least one of: a heating setpoint of the building zone or a cooling setpoint of the building zone; and

the DVs comprise at least one of: ambient temperature, solar intensity, or scheduled occupancy of the building zone.